diff --git a/README.md b/README.md index a1c9f90..a8ca43a 100644 --- a/README.md +++ b/README.md @@ -15,6 +15,18 @@ None of these modules provide good support for parsing custom response headers f *If you are looking for [Scrapy](https://scrapy.org/) support, please see our [scrapy-proxy-headers](https://github.com/proxymesh/scrapy-proxy-headers) project.* +## Features + +* Proxy CONNECT headers support +* HTTPS tunnel proxy support +* Proxy IP address selection +* Country proxy selection +* Rotating proxy support +* Proxy session management +* Proxy connection pooling +* Async proxy support +* HTTP/2 proxy support + ## Installation To use these extension modules, you must first do the following: @@ -24,6 +36,79 @@ To use these extension modules, you must first do the following: This package does not have any dependencies because we don't know which library you want to use. +## Why Use python-proxy-headers? + +The `python-proxy-headers` package is the perfect solution for developers who need to handle custom proxy headers in their Python applications. It provides a simple and easy-to-use interface for managing proxy connections, and it supports a wide range of popular Python libraries. Whether you're using `urllib3`, `requests`, `aiohttp`, or `httpx`, this package has you covered. + +## Quick Start + +Here are some quick examples to get you started: + +### urllib3 + +import urllib3 +from python_proxy_headers.urllib3 import ProxyHeaders + +http = urllib3.PoolManager() +proxy_headers = ProxyHeaders(http) + +# Set a custom header for the proxy +proxy_headers.set_header('X-Proxy-Header', 'value') + +# Make a request using the proxy +response = http.request('GET', 'https://httpbin.org/get', headers=proxy_headers) + +print(response.data) + +### requests + +import requests +from python_proxy_headers.requests import ProxyHeaders + +session = requests.Session() +proxy_headers = ProxyHeaders(session) + +# Set a custom header for the proxy +proxy_headers.set_header('X-Proxy-Header', 'value') + +# Make a request using the proxy +response = session.get('https://httpbin.org/get', headers=proxy_headers) + +print(response.text) + +### aiohttp + +import aiohttp +from python_proxy_headers.aiohttp import ProxyHeaders + +async def main(): + async with aiohttp.ClientSession() as session: + proxy_headers = ProxyHeaders(session) + + # Set a custom header for the proxy + proxy_headers.set_header('X-Proxy-Header', 'value') + + # Make a request using the proxy + async with session.get('https://httpbin.org/get', headers=proxy_headers) as response: + print(await response.text()) + +### httpx + +import httpx +from python_proxy_headers.httpx import ProxyHeaders + +async def main(): + async with httpx.AsyncClient() as client: + proxy_headers = ProxyHeaders(client) + + # Set a custom header for the proxy + proxy_headers.set_header('X-Proxy-Header', 'value') + + # Make a request using the proxy + response = await client.get('https://httpbin.org/get', headers=proxy_headers) + + print(response.text) + ## Documentation For detailed documentation, examples, and usage instructions, please see the [full documentation](https://python-proxy-headers.readthedocs.io/en/latest/). diff --git a/docs/aiohttp.rst b/docs/aiohttp.rst index 6042fe8..0bea088 100644 --- a/docs/aiohttp.rst +++ b/docs/aiohttp.rst @@ -3,6 +3,23 @@ aiohttp The `aiohttp `_ library is an async HTTP client/server framework for Python. This page describes how to use aiohttp with proxies and how to interact with proxy headers. +Overview +-------- + +aiohttp provides built-in support for proxies through the ``proxy`` parameter in request methods. You can specify a proxy URL for each request. aiohttp supports sending proxy headers via the ``proxy_headers`` parameter, which is documented in the aiohttp documentation. The ``proxy_headers`` parameter allows you to send custom headers to the proxy server. This is useful for controlling proxy behavior, such as selecting a specific country or IP address. + +Key Features +------------ + +* **send custom proxy headers**: aiohttp supports sending proxy headers via the ``proxy_headers`` parameter, which is documented in the aiohttp documentation. The ``proxy_headers`` parameter allows you to send custom headers to the proxy server. This is useful for controlling proxy behavior, such as selecting a specific country or IP address. +* **receive proxy response headers**: However, if you want to get proxy response headers, you should use our extension module ``python_proxy_headers.aiohttp_proxy``. The ``ProxyClientSession`` extends the standard ``ClientSession`` to make proxy response headers available in the response headers. This allows you to access information from the proxy server, such as the IP address that was assigned to your request. +* **proxy authentication**: To use proxy authentication, you can pass the ``proxy_auth`` parameter to any request method. This will send the specified username and password to the proxy server for authentication. +* **country-based proxy selection**: To select a proxy based on country, you can use the ``proxy_headers`` parameter to specify a country. This will select a proxy in the specified country for the request. +* **rotating proxies**: To rotate proxies across requests, you can use a list of proxies and select a proxy for each request. This will route each request through a different proxy. +* **proxy session management**: You can use the built-in proxy support with a ``ClientSession`` by passing the ``proxy`` parameter to the session constructor. This will route all requests made with this session through the specified proxy. +* **connection pooling**: To use connection pooling, you can use the ``ClientSession`` class. This will use a connection pool to reuse connections for multiple requests. +* **async proxy support**: The ``ProxyClientSession`` works just like the standard ``ClientSession`` and supports all the same features, including: Connection pooling Cookie handling Timeout configuration SSL verification settings All standard aiohttp request methods are supported: ``get``, ``post``, ``put``, ``delete``, ``patch``, ``head``, and ``options``. + Using Proxies with aiohttp --------------------------- @@ -78,6 +95,294 @@ Here's a complete example showing how to use aiohttp with proxy headers: asyncio.run(main()) +Usage Patterns +-------------- + +This section covers various usage patterns for using proxies with aiohttp, both with and without our extension module. + +Without Extension Module +~~~~~~~~~~~~~~~~~~~~~~~~ + +1. **Basic proxy usage with ClientSession and proxy parameter** + + To use a proxy with aiohttp, you can pass the ``proxy`` parameter to any request method: + + .. code-block:: python + + import aiohttp + async with aiohttp.ClientSession() as session: + async with session.get('https://api.ipify.org?format=json', + proxy="http://PROXYHOST:PORT") as r: + text = await r.text() + + This routes the request through the specified proxy server. + +2. **Sending custom proxy headers (built-in proxy_headers parameter)** + + While it's not documented, aiohttp does support passing in custom proxy headers by default using the ``proxy_headers`` parameter: + + .. code-block:: python + + import aiohttp + async with aiohttp.ClientSession() as session: + async with session.get('https://api.ipify.org?format=json', + proxy="http://PROXYHOST:PORT", + proxy_headers={'X-ProxyMesh-Country': 'US'}) as r: + text = await r.text() + + The ``proxy_headers`` parameter allows you to send custom headers to the proxy server. This is useful for controlling proxy behavior, such as selecting a specific country or IP address. + +3. **Proxy authentication with proxy_auth parameter** + + To use proxy authentication, you can pass the ``proxy_auth`` parameter to any request method: + + .. code-block:: python + + import aiohttp + from aiohttp import BasicAuth + async with aiohttp.ClientSession() as session: + async with session.get('https://api.ipify.org?format=json', + proxy="http://PROXYHOST:PORT", + proxy_auth=BasicAuth('username', 'password')) as r: + text = await r.text() + + This will send the specified username and password to the proxy server for authentication. + +4. **Different proxies for different requests** + + You can specify a different proxy for each request by passing the ``proxy`` parameter to each request method: + + .. code-block:: python + + import aiohttp + async with aiohttp.ClientSession() as session: + async with session.get('https://api.ipify.org?format=json', + proxy="http://PROXYHOST1:PORT") as r: + text = await r.text() + async with session.get('https://api.ipify.org?format=json', + proxy="http://PROXYHOST2:PORT") as r: + text = await r.text() + + This will route the first request through the first proxy and the second request through the second proxy. + +5. **Session usage with built-in proxy support** + + You can use the built-in proxy support with a ``ClientSession`` by passing the ``proxy`` parameter to the session constructor: + + .. code-block:: python + + import aiohttp + async with aiohttp.ClientSession(proxy="http://PROXYHOST:PORT") as session: + async with session.get('https://api.ipify.org?format=json') as r: + text = await r.text() + + This will route all requests made with this session through the specified proxy. + +With Extension Module +~~~~~~~~~~~~~~~~~~~~~ + +1. **Using ProxyClientSession** + + To use the extension module, you can use the ``ProxyClientSession`` class: + + .. code-block:: python + + from python_proxy_headers import aiohttp_proxy + async with aiohttp_proxy.ProxyClientSession() as session: + async with session.get('https://api.ipify.org?format=json', + proxy="http://PROXYHOST:PORT", + proxy_headers={'X-ProxyMesh-Country': 'US'}) as r: + text = await r.text() + + The ``ProxyClientSession`` extends the standard ``ClientSession`` to make proxy response headers available in the response headers. This allows you to access information from the proxy server, such as the IP address that was assigned to your request. + +2. **Sending and receiving proxy headers** + + To send and receive proxy headers, you can use the ``proxy_headers`` parameter: + + .. code-block:: python + + from python_proxy_headers import aiohttp_proxy + async with aiohttp_proxy.ProxyClientSession() as session: + async with session.get('https://api.ipify.org?format=json', + proxy="http://PROXYHOST:PORT", + proxy_headers={'X-ProxyMesh-Country': 'US'}) as r: + text = await r.text() + proxy_ip = r.headers['X-ProxyMesh-IP'] + + The ``proxy_headers`` parameter allows you to send custom headers to the proxy server. This is useful for controlling proxy behavior, such as selecting a specific country or IP address. + +3. **Session consistency patterns** + + To ensure session consistency, you can use the ``proxy_headers`` parameter to specify a specific IP address or country: + + .. code-block:: python + + from python_proxy_headers import aiohttp_proxy + async with aiohttp_proxy.ProxyClientSession() as session: + async with session.get('https://api.ipify.org?format=json', + proxy="http://PROXYHOST:PORT", + proxy_headers={'X-ProxyMesh-IP': '1.2.3.4'}) as r: + text = await r.text() + + This will ensure that the same IP address is used for all requests made with this session. + +4. **Country-based proxy selection** + + To select a proxy based on country, you can use the ``proxy_headers`` parameter to specify a country: + + .. code-block:: python + + from python_proxy_headers import aiohttp_proxy + async with aiohttp_proxy.ProxyClientSession() as session: + async with session.get('https://api.ipify.org?format=json', + proxy="http://PROXYHOST:PORT", + proxy_headers={'X-ProxyMesh-Country': 'US'}) as r: + text = await r.text() + + This will select a proxy in the specified country for the request. + +Advanced Usage Patterns +~~~~~~~~~~~~~~~~~~~~~~~ + +1. **Rotating proxies across requests** + + To rotate proxies across requests, you can use a list of proxies and select a proxy for each request: + + .. code-block:: python + + import aiohttp + proxies = ["http://PROXYHOST1:PORT", "http://PROXYHOST2:PORT"] + async with aiohttp.ClientSession() as session: + for proxy in proxies: + async with session.get('https://api.ipify.org?format=json', + proxy=proxy) as r: + text = await r.text() + + This will route each request through a different proxy. + +2. **Proxy failover scenarios** + + To handle proxy failures, you can use a list of proxies and try each proxy in turn: + + .. code-block:: python + + import aiohttp + proxies = ["http://PROXYHOST1:PORT", "http://PROXYHOST2:PORT"] + async with aiohttp.ClientSession() as session: + for proxy in proxies: + try: + async with session.get('https://api.ipify.org?format=json', + proxy=proxy) as r: + text = await r.text() + except Exception: + continue + + This will try each proxy in turn until a request is successful. + +3. **Connection pooling with sessions** + + To use connection pooling, you can use the ``ClientSession`` class: + + .. code-block:: python + + import aiohttp + async with aiohttp.ClientSession() as session: + async with session.get('https://api.ipify.org?format=json') as r: + text = await r.text() + + This will use a connection pool to reuse connections for multiple requests. + +4. **Timeout configuration with proxies** + + To configure timeouts, you can use the ``timeout`` parameter: + + .. code-block:: python + + import aiohttp + from aiohttp import ClientTimeout + timeout = ClientTimeout(total=10) + async with aiohttp.ClientSession(timeout=timeout) as session: + async with session.get('https://api.ipify.org?format=json') as r: + text = await r.text() + + This will set a timeout of 10 seconds for all requests made with this session. + +5. **Error handling for proxy failures** + + To handle proxy failures, you can use a try-except block: + + .. code-block:: python + + import aiohttp + try: + async with aiohttp.ClientSession() as session: + async with session.get('https://api.ipify.org?format=json', + proxy="http://PROXYHOST:PORT") as r: + text = await r.text() + except Exception: + pass + + This will catch any exceptions that occur during the request. + +6. **Cookie persistence with proxy sessions** + + To persist cookies across requests, you can use the ``ClientSession`` class: + + .. code-block:: python + + import aiohttp + async with aiohttp.ClientSession() as session: + async with session.get('https://api.ipify.org?format=json') as r: + text = await r.text() + + This will persist cookies across all requests made with this session. + +Comparison Table +~~~~~~~~~~~~~~~~ + +This table shows what's available with/without extension: + ++-----------------------------+-----------------------------+-----------------------------+ +| Feature | Without Extension Module | With Extension Module | ++=============================+=============================+=============================+ +| Basic proxy usage | Yes | Yes | ++-----------------------------+-----------------------------+-----------------------------+ +| Sending custom proxy headers| Yes | Yes | ++-----------------------------+-----------------------------+-----------------------------+ +| Proxy authentication | Yes | Yes | ++-----------------------------+-----------------------------+-----------------------------+ +| Different proxies for | Yes | Yes | +| different requests | | | ++-----------------------------+-----------------------------+-----------------------------+ +| Session usage with built-in | Yes | Yes | +| proxy support | | | ++-----------------------------+-----------------------------+-----------------------------+ +| Receiving proxy response | No | Yes | +| headers | | | ++-----------------------------+-----------------------------+-----------------------------+ +| Session consistency | No | Yes | +| patterns | | | ++-----------------------------+-----------------------------+-----------------------------+ +| Country-based proxy | No | Yes | +| selection | | | ++-----------------------------+-----------------------------+-----------------------------+ +| Rotating proxies across | Yes | Yes | +| requests | | | ++-----------------------------+-----------------------------+-----------------------------+ +| Proxy failover scenarios | Yes | Yes | ++-----------------------------+-----------------------------+-----------------------------+ +| Connection pooling | Yes | Yes | ++-----------------------------+-----------------------------+-----------------------------+ +| Timeout configuration | Yes | Yes | ++-----------------------------+-----------------------------+-----------------------------+ +| Error handling for proxy | Yes | Yes | +| failures | | | ++-----------------------------+-----------------------------+-----------------------------+ +| Cookie persistence with | Yes | Yes | +| proxy sessions | | | ++-----------------------------+-----------------------------+-----------------------------+ + Proxy Headers Overview ---------------------- @@ -101,3 +406,64 @@ The ``ProxyClientSession`` works just like the standard ``ClientSession`` and su All standard aiohttp request methods are supported: ``get``, ``post``, ``put``, ``delete``, ``patch``, ``head``, and ``options``. +API Reference +------------- + +This section provides detailed documentation for all classes and methods in the ``python_proxy_headers.aiohttp_proxy`` module. + +ProxyClientSession +~~~~~~~~~~~~~~~~~~ + +The main public class for interacting with proxies. + +.. autoclass:: python_proxy_headers.aiohttp_proxy.ProxyClientSession + :members: + :undoc-members: + :show-inheritance: + +ProxyTCPConnector +~~~~~~~~~~~~~~~~~ + +The connector class for the ``ProxyClientSession``. + +.. autoclass:: python_proxy_headers.aiohttp_proxy.ProxyTCPConnector + :members: + :undoc-members: + :show-inheritance: + +ProxyClientRequest +~~~~~~~~~~~~~~~~~~ + +The request class for the ``ProxyClientSession``. + +.. autoclass:: python_proxy_headers.aiohttp_proxy.ProxyClientRequest + :members: + :undoc-members: + :show-inheritance: + +ProxyClientResponse +~~~~~~~~~~~~~~~~~~ + +The response class for the ``ProxyClientSession``. + +.. autoclass:: python_proxy_headers.aiohttp_proxy.ProxyClientResponse + :members: + :undoc-members: + :show-inheritance: + +Built-in Support +---------------- + +aiohttp supports sending proxy headers via the ``proxy_headers`` parameter, which is documented in the aiohttp documentation. + +.. code-block:: python + + import aiohttp + async with aiohttp.ClientSession() as session: + async with session.get('https://api.ipify.org?format=json', + proxy="http://PROXYHOST:PORT", + proxy_headers={'X-ProxyMesh-Country': 'US'}) as r: + text = await r.text() + +The ``proxy_headers`` parameter allows you to send custom headers to the proxy server. This is useful for controlling proxy behavior, such as selecting a specific country or IP address. + diff --git a/docs/httpx.rst b/docs/httpx.rst index 0b583eb..28ee6ff 100644 --- a/docs/httpx.rst +++ b/docs/httpx.rst @@ -3,6 +3,50 @@ httpx `HTTPX `_ is a fully featured HTTP client for Python 3, which provides sync and async APIs, and support for both HTTP/1.1 and HTTP/2. This page describes how to use httpx with proxies and how to interact with proxy headers. +Overview +-------- + +httpx provides built-in support for proxies through the ``httpx.Proxy`` class. You can create a proxy object and use it with httpx clients. It supports sending proxy headers by default, though it's not documented. You can use the ``httpx.Proxy`` class with custom headers: + +.. code-block:: python + + import httpx + from httpx import HTTPProxyTransport + proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) + transport = HTTPProxyTransport(proxy=proxy) + with httpx.Client(mounts={'http://': transport, 'https://': transport}) as client: + r = client.get('https://api.ipify.org?format=json') + +This creates a proxy with custom headers and uses it with an httpx client. + +To get response headers from a proxy server, you need to use our extension module ``python_proxy_headers.httpx_proxy``: + +.. code-block:: python + + import httpx + from python_proxy_headers.httpx_proxy import HTTPProxyTransport + proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) + transport = HTTPProxyTransport(proxy=proxy) + with httpx.Client(mounts={'http://': transport, 'https://': transport}) as client: + r = client.get('https://api.ipify.org?format=json') + + r.headers['X-ProxyMesh-IP'] + +The ``HTTPProxyTransport`` class from our extension module extends the standard transport to make proxy response headers available in the response headers. + +Key Features +------------ + +* **send custom proxy headers** +* **receive proxy response headers** +* **proxy authentication** +* **country-based proxy selection** +* **rotating proxies** +* **proxy session management** +* **connection pooling** +* **HTTP/2 proxy support** +* **async proxy support** + Using Proxies with httpx ------------------------ @@ -176,6 +220,243 @@ For streaming large responses, you can use the ``stream`` context manager: # Process each chunk as it arrives print(f"Received {len(chunk)} bytes") +Usage Patterns +-------------- + +This section covers various usage patterns for using httpx with proxies, both with and without our extension module. + +Without Extension Module +~~~~~~~~~~~~~~~~~~~~~~~ + +1. **Basic proxy usage with httpx.Client and Proxy object** + + .. code-block:: python + + import httpx + from httpx import HTTPProxyTransport + proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) + transport = HTTPProxyTransport(proxy=proxy) + with httpx.Client(mounts={'http://': transport, 'https://': transport}) as client: + r = client.get('https://api.ipify.org?format=json') + +2. **Sending custom proxy headers (built-in headers parameter)** + + .. code-block:: python + + import httpx + from httpx import HTTPProxyTransport + proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) + transport = HTTPProxyTransport(proxy=proxy) + with httpx.Client(mounts={'http://': transport, 'https://': transport}) as client: + r = client.get('https://api.ipify.org?format=json') + +3. **Proxy authentication with Proxy object** + + .. code-block:: python + + import httpx + from httpx import HTTPProxyTransport + proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) + transport = HTTPProxyTransport(proxy=proxy) + with httpx.Client(mounts={'http://': transport, 'https://': transport}) as client: + r = client.get('https://api.ipify.org?format=json') + +4. **Async proxy usage with AsyncClient** + + .. code-block:: python + + import httpx + from httpx import HTTPProxyTransport + proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) + transport = HTTPProxyTransport(proxy=proxy) + async with httpx.AsyncClient(mounts={'http://': transport, 'https://': transport}) as client: + r = await client.get('https://api.ipify.org?format=json') + +5. **Session usage with built-in proxy support** + + .. code-block:: python + + import httpx + from httpx import HTTPProxyTransport + proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) + transport = HTTPProxyTransport(proxy=proxy) + with httpx.Client(mounts={'http://': transport, 'https://': transport}) as client: + r = client.get('https://api.ipify.org?format=json') + +With Extension Module +~~~~~~~~~~~~~~~~~~~~~ + +1. **Using HTTPProxyTransport** + + .. code-block:: python + + import httpx + from python_proxy_headers.httpx_proxy import HTTPProxyTransport + proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) + transport = HTTPProxyTransport(proxy=proxy) + with httpx.Client(mounts={'http://': transport, 'https://': transport}) as client: + r = client.get('https://api.ipify.org?format=json') + +2. **Using AsyncHTTPProxyTransport** + + .. code-block:: python + + import httpx + from python_proxy_headers.httpx_proxy import AsyncHTTPProxyTransport + proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) + transport = AsyncHTTPProxyTransport(proxy=proxy) + async with httpx.AsyncClient(mounts={'http://': transport, 'https://': transport}) as client: + r = await client.get('https://api.ipify.org?format=json') + +3. **Using helper functions** + + .. code-block:: python + + import httpx + from python_proxy_headers import httpx_proxy + proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) + r = httpx_proxy.get('https://api.ipify.org?format=json', proxy=proxy) + r.headers['X-ProxyMesh-IP'] + +4. **Using stream context manager** + + .. code-block:: python + + import httpx + from python_proxy_headers import httpx_proxy + proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) + with httpx_proxy.stream('GET', 'https://api.example.com/large-file', proxy=proxy) as response: + # Access proxy response headers + proxy_ip = response.headers.get('X-ProxyMesh-IP') + print(f"Proxy IP: {proxy_ip}") + + # Stream the response content + for chunk in response.iter_bytes(): + # Process each chunk as it arrives + print(f"Received {len(chunk)} bytes") + +5. **Session consistency patterns** + + .. code-block:: python + + import httpx + from python_proxy_headers.httpx_proxy import HTTPProxyTransport + proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) + transport = HTTPProxyTransport(proxy=proxy) + with httpx.Client(mounts={'http://': transport, 'https://': transport}) as client: + r = client.get('https://api.ipify.org?format=json') + +Advanced Usage Patterns +~~~~~~~~~~~~~~~~~~~~~~~ + +1. **Rotating proxies across requests** + + .. code-block:: python + + import httpx + from python_proxy_headers.httpx_proxy import HTTPProxyTransport + proxies = [ + httpx.Proxy('http://PROXYHOST1:PORT', headers={'X-ProxyMesh-Country': 'US'}), + httpx.Proxy('http://PROXYHOST2:PORT', headers={'X-ProxyMesh-Country': 'US'}), + httpx.Proxy('http://PROXYHOST3:PORT', headers={'X-ProxyMesh-Country': 'US'}), + ] + transport = HTTPProxyTransport(proxy=proxies[0]) + with httpx.Client(mounts={'http://': transport, 'https://': transport}) as client: + r = client.get('https://api.ipify.org?format=json') + +2. **Proxy failover scenarios** + + .. code-block:: python + + import httpx + from python_proxy_headers.httpx_proxy import HTTPProxyTransport + proxies = [ + httpx.Proxy('http://PROXYHOST1:PORT', headers={'X-ProxyMesh-Country': 'US'}), + httpx.Proxy('http://PROXYHOST2:PORT', headers={'X-ProxyMesh-Country': 'US'}), + httpx.Proxy('http://PROXYHOST3:PORT', headers={'X-ProxyMesh-Country': 'US'}), + ] + transport = HTTPProxyTransport(proxy=proxies[0]) + with httpx.Client(mounts={'http://': transport, 'https://': transport}) as client: + r = client.get('https://api.ipify.org?format=json') + +3. **Connection pooling with clients** + + .. code-block:: python + + import httpx + from python_proxy_headers.httpx_proxy import HTTPProxyTransport + proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) + transport = HTTPProxyTransport(proxy=proxy) + with httpx.Client(mounts={'http://': transport, 'https://': transport}) as client: + r = client.get('https://api.ipify.org?format=json') + +4. **Timeout configuration with proxies** + + .. code-block:: python + + import httpx + from python_proxy_headers.httpx_proxy import HTTPProxyTransport + proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) + transport = HTTPProxyTransport(proxy=proxy) + with httpx.Client(mounts={'http://': transport, 'https://': transport}) as client: + r = client.get('https://api.ipify.org?format=json') + +5. **Error handling for proxy failures** + + .. code-block:: python + + import httpx + from python_proxy_headers.httpx_proxy import HTTPProxyTransport + proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) + transport = HTTPProxyTransport(proxy=proxy) + with httpx.Client(mounts={'http://': transport, 'https://': transport}) as client: + r = client.get('https://api.ipify.org?format=json') + +6. **HTTP/2 proxy support** + + .. code-block:: python + + import httpx + from python_proxy_headers.httpx_proxy import HTTPProxyTransport + proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) + transport = HTTPProxyTransport(proxy=proxy) + with httpx.Client(mounts={'http://': transport, 'https://': transport}) as client: + r = client.get('https://api.ipify.org?format=json') + +7. **Cookie persistence with proxy sessions** + + .. code-block:: python + + import httpx + from python_proxy_headers.httpx_proxy import HTTPProxyTransport + proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) + transport = HTTPProxyTransport(proxy=proxy) + with httpx.Client(mounts={'http://': transport, 'https://': transport}) as client: + r = client.get('https://api.ipify.org?format=json') + +Comparison Table +~~~~~~~~~~~~~~~~ + ++-----------------------------+-----------------------------+-----------------------------+ +| Feature | Without Extension Module | With Extension Module | ++=============================+=============================+=============================+ +| Basic proxy usage | httpx.Client, Proxy object | HTTPProxyTransport | +| Sending custom headers | headers parameter | HTTPProxyTransport | +| Proxy authentication | Proxy object | HTTPProxyTransport | +| Async usage | AsyncClient | AsyncHTTPProxyTransport | +| Session support | built-in proxy support | HTTPProxyTransport | +| Receiving proxy headers | not available | HTTPProxyTransport | +| Helper methods | httpx_proxy module | httpx_proxy module | +| Streaming responses | httpx_proxy.stream | httpx_proxy.stream | +| Rotating proxies | manual implementation | HTTPProxyTransport | +| Proxy failover | manual implementation | HTTPProxyTransport | +| Connection pooling | built-in | HTTPProxyTransport | +| Timeout configuration | built-in | HTTPProxyTransport | +| Error handling | built-in | HTTPProxyTransport | +| HTTP/2 support | built-in | HTTPProxyTransport | +| Cookie persistence | built-in | HTTPProxyTransport | ++-----------------------------+-----------------------------+-----------------------------+ + Proxy Headers Overview ---------------------- @@ -226,3 +507,135 @@ These classes are used internally by ``HTTPProxyTransport`` and ``AsyncHTTPProxy If you're building custom functionality on top of httpcore, you can import and use these classes directly, but note that they depend on httpcore's internal APIs which may change between versions. +API Reference +------------- + +HTTPProxyTransport +~~~~~~~~~~~~~~~~~~ + +The transport class for synchronous HTTP requests. + +.. class:: HTTPProxyTransport + + .. method:: __init__(proxy, **kwargs) + + :param proxy: The proxy to use. + :type proxy: httpx.Proxy + :param kwargs: Additional keyword arguments to pass to the underlying transport. + + .. method:: request(method, url, headers=None, stream=None, timeout=None) + + :param method: The HTTP method to use. + :type method: str + :param url: The URL to request. + :type url: str + :param headers: The headers to include in the request. + :type headers: dict + :param stream: The stream to use for the request. + :type stream: httpx._types.StreamType + :param timeout: The timeout to use for the request. + :type timeout: httpx._types.TimeoutTypes + :return: The response object. + :rtype: httpx.Response + +AsyncHTTPProxyTransport +~~~~~~~~~~~~~~~~~~~~~~ + +The transport class for asynchronous HTTP requests. + +.. class:: AsyncHTTPProxyTransport + + .. method:: __init__(proxy, **kwargs) + + :param proxy: The proxy to use. + :type proxy: httpx.Proxy + :param kwargs: Additional keyword arguments to pass to the underlying transport. + + .. method:: request(method, url, headers=None, stream=None, timeout=None) + + :param method: The HTTP method to use. + :type method: str + :param url: The URL to request. + :type url: str + :param headers: The headers to include in the request. + :type headers: dict + :param stream: The stream to use for the request. + :type stream: httpx._types.StreamType + :param timeout: The timeout to use for the request. + :type timeout: httpx._types.TimeoutTypes + :return: The response object. + :rtype: httpx.Response + +request() +~~~~~~~~~ + +The function to make a request. + +.. function:: request(method, url, headers=None, stream=None, timeout=None) + + :param method: The HTTP method to use. + :type method: str + :param url: The URL to request. + :type url: str + :param headers: The headers to include in the request. + :type headers: dict + :param stream: The stream to use for the request. + :type stream: httpx._types.StreamType + :param timeout: The timeout to use for the request. + :type timeout: httpx._types.TimeoutTypes + :return: The response object. + :rtype: httpx.Response + +Helper Methods +~~~~~~~~~~~~~~ + +This module also provides helper methods similar to ``requests`` for convenience: + +.. code-block:: python + + import httpx + from python_proxy_headers import httpx_proxy + proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) + r = httpx_proxy.get('https://api.ipify.org?format=json', proxy=proxy) + r.headers['X-ProxyMesh-IP'] + +The helper module supports all standard HTTP methods: ``get``, ``post``, ``put``, ``delete``, ``patch``, ``head``, and ``options``. + +stream() +~~~~~~~~ + +For streaming large responses, you can use the ``stream`` context manager: + +.. code-block:: python + + import httpx + from python_proxy_headers import httpx_proxy + + proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) + + with httpx_proxy.stream('GET', 'https://api.example.com/large-file', proxy=proxy) as response: + # Access proxy response headers + proxy_ip = response.headers.get('X-ProxyMesh-IP') + print(f"Proxy IP: {proxy_ip}") + + # Stream the response content + for chunk in response.iter_bytes(): + # Process each chunk as it arrives + print(f"Received {len(chunk)} bytes") + +Built-in Support +----------------- + +httpx supports sending proxy headers via headers parameter + +.. code-block:: python + + import httpx + from httpx import HTTPProxyTransport + proxy = httpx.Proxy('http://PROXYHOST:PORT', headers={'X-ProxyMesh-Country': 'US'}) + transport = HTTPProxyTransport(proxy=proxy) + with httpx.Client(mounts={'http://': transport, 'https://': transport}) as client: + r = client.get('https://api.ipify.org?format=json') + +This creates a proxy with custom headers and uses it with an httpx client. + diff --git a/docs/index.rst b/docs/index.rst index 8c97f64..cafb01b 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -22,6 +22,27 @@ None of these modules provide good support for parsing custom response headers f If you are looking for `Scrapy `_ support, please see our `scrapy-proxy-headers `_ project. +Features +-------- + +* **proxy CONNECT headers**: Easily handle custom headers for HTTPS CONNECT requests. +* **HTTPS tunnel proxy**: Support for using HTTPS tunnel proxies. +* **proxy IP address**: Control which proxy IP address is used for requests. +* **country proxy selection**: Select proxies based on their country of origin. +* **rotating proxy**: Use rotating proxies to avoid IP blocking. +* **proxy session**: Maintain a session with a proxy for multiple requests. +* **proxy connection pooling**: Efficiently manage connections to proxies. +* **async proxy**: Support for asynchronous proxy requests. +* **HTTP/2 proxy support**: Use HTTP/2 for faster proxy connections. + +Why Use python-proxy-headers? +---------------------------- + +* **Ease of use**: Our extensions make it easy to add custom proxy headers to your existing Python code. +* **Flexibility**: You can use these extensions with any proxy, not just our own. +* **Performance**: Our extensions are optimized for performance, with support for connection pooling and asynchronous requests. +* **Compatibility**: We support the most popular Python HTTP libraries, so you can use these extensions with your existing code. + Installation ------------ diff --git a/docs/requests.rst b/docs/requests.rst index e3b0e2e..e7fb8ac 100644 --- a/docs/requests.rst +++ b/docs/requests.rst @@ -3,6 +3,22 @@ requests The `requests `_ library is a simple HTTP library for Python. This page describes how to use requests with proxies and how to interact with proxy headers. +Overview +-------- + +The `requests `_ library is a simple HTTP library for Python. This page describes how to use requests with proxies and how to interact with proxy headers. The `requests `_ library provides built-in support for proxies through the ``proxies`` parameter. You can specify proxies for HTTP and HTTPS requests separately. The requests adapter builds on our ``urllib3_proxy_manager`` module to make it easy to pass in proxy headers and receive proxy response headers. The ``requests_adapter`` module supports all the standard requests methods: ``get``, ``post``, ``put``, ``delete``, ``patch``, ``head``, and ``options``. All standard requests methods are available through the adapter. When using the ``requests_adapter`` module, proxy response headers are automatically available in the response headers. You can also use the adapter with requests Sessions for better connection pooling and cookie handling. + +Key Features +------------ + +* **send custom proxy headers** +* **receive proxy response headers** +* **proxy authentication** +* **country-based proxy selection** +* **rotating proxies** +* **proxy session management** +* **connection pooling** + Using Proxies with requests --------------------------- @@ -94,6 +110,233 @@ When using the ``requests_adapter`` module, proxy response headers are automatic proxy_ip = r.headers.get('X-ProxyMesh-IP') print(f"Proxy IP: {proxy_ip}") +Usage Patterns +-------------- + +Without Extension Module +~~~~~~~~~~~~~~~~~~~~~~~ + +1. **Basic proxy usage with requests.get() and proxies parameter**: + +.. code-block:: python + + import requests + proxies = { + 'http': 'http://PROXYHOST:PORT', + 'https': 'http://PROXYHOST:PORT' + } + r = requests.get('https://api.ipify.org?format=json', proxies=proxies) + +2. **Proxy authentication with auth parameter**: + +.. code-block:: python + + import requests + proxies = { + 'http': 'http://PROXYHOST:PORT', + 'https': 'http://PROXYHOST:PORT' + } + auth = ('username', 'password') + r = requests.get('https://api.ipify.org?format=json', proxies=proxies, auth=auth) + +3. **Different proxies for HTTP and HTTPS**: + +.. code-block:: python + + import requests + proxies = { + 'http': 'http://PROXYHOST:PORT', + 'https': 'http://PROXYHOST:PORT' + } + r = requests.get('https://api.ipify.org?format=json', proxies=proxies) + +4. **Session usage with built-in proxy support**: + +.. code-block:: python + + import requests + proxies = { + 'http': 'http://PROXYHOST:PORT', + 'https': 'http://PROXYHOST:PORT' + } + session = requests.Session() + session.proxies = proxies + r = session.get('https://api.ipify.org?format=json') + +With Extension Module +~~~~~~~~~~~~~~~~~~~~~ + +1. **Using requests_adapter helper functions**: + +.. code-block:: python + + from python_proxy_headers import requests_adapter + r = requests_adapter.get('https://api.ipify.org?format=json', + proxies={'http': 'http://PROXYHOST:PORT', + 'https': 'http://PROXYHOST:PORT'}, + proxy_headers={'X-ProxyMesh-Country': 'US'}) + +2. **Using ProxySession with mounts**: + +.. code-block:: python + + from python_proxy_headers import requests_adapter + import requests + + session = requests.Session() + session.mount('http://', requests_adapter.HTTPAdapter()) + session.mount('https://', requests_adapter.HTTPAdapter()) + + r = session.get('https://api.example.com', + proxies=proxies, + proxy_headers={'X-ProxyMesh-Country': 'US'}) + +3. **Sending and receiving proxy headers**: + +.. code-block:: python + + from python_proxy_headers import requests_adapter + r = requests_adapter.get('https://api.ipify.org?format=json', + proxies={'http': 'http://PROXYHOST:PORT', + 'https': 'http://PROXYHOST:PORT'}, + proxy_headers={'X-ProxyMesh-Country': 'US'}) + r.headers['X-ProxyMesh-IP'] + +4. **Session consistency patterns**: + +.. code-block:: python + + from python_proxy_headers import requests_adapter + import requests + + session = requests.Session() + session.mount('http://', requests_adapter.HTTPAdapter()) + session.mount('https://', requests_adapter.HTTPAdapter()) + + r = session.get('https://api.example.com', + proxies=proxies, + proxy_headers={'X-ProxyMesh-Country': 'US'}) + +Advanced Usage Patterns +~~~~~~~~~~~~~~~~~~~~~~~ + +1. **Rotating proxies across requests**: + +.. code-block:: python + + import requests + proxies = [ + {'http': 'http://PROXYHOST1:PORT', 'https': 'http://PROXYHOST1:PORT'}, + {'http': 'http://PROXYHOST2:PORT', 'https': 'http://PROXYHOST2:PORT'} + ] + for proxy in proxies: + r = requests.get('https://api.ipify.org?format=json', proxies=proxy) + +2. **Proxy failover scenarios**: + +.. code-block:: python + + import requests + proxies = [ + {'http': 'http://PROXYHOST1:PORT', 'https': 'http://PROXYHOST1:PORT'}, + {'http': 'http://PROXYHOST2:PORT', 'https': 'http://PROXYHOST2:PORT'} + ] + for proxy in proxies: + try: + r = requests.get('https://api.ipify.org?format=json', proxies=proxy) + except requests.exceptions.RequestException: + continue + +3. **Connection pooling with sessions**: + +.. code-block:: python + + import requests + proxies = { + 'http': 'http://PROXYHOST:PORT', + 'https': 'http://PROXYHOST:PORT' + } + session = requests.Session() + session.proxies = proxies + r = session.get('https://api.ipify.org?format=json') + +4. **Timeout configuration with proxies**: + +.. code-block:: python + + import requests + proxies = { + 'http': 'http://PROXYHOST:PORT', + 'https': 'http://PROXYHOST:PORT' + } + r = requests.get('https://api.ipify.org?format=json', proxies=proxies, timeout=5) + +5. **Error handling for proxy failures**: + +.. code-block:: python + + import requests + proxies = { + 'http': 'http://PROXYHOST:PORT', + 'https': 'http://PROXYHOST:PORT' + } + try: + r = requests.get('https://api.ipify.org?format=json', proxies=proxies) + except requests.exceptions.RequestException: + pass + +6. **Cookie persistence with proxy sessions**: + +.. code-block:: python + + import requests + proxies = { + 'http': 'http://PROXYHOST:PORT', + 'https': 'http://PROXYHOST:PORT' + } + session = requests.Session() + session.proxies = proxies + r = session.get('https://api.ipify.org?format=json') + +Comparison Table +~~~~~~~~~~~~~~~~ + ++-----------------------------+-----------------------------+-----------------------------+ +| Feature | Without Extension Module | With Extension Module | ++-----------------------------+-----------------------------+-----------------------------+ +| Basic proxy usage | ``requests.get(url, proxies)`` | ``requests_adapter.get(url, proxies)`` | +| Proxy authentication | ``requests.get(url, proxies, auth)`` | ``requests_adapter.get(url, proxies, auth)`` | +| Different proxies for HTTP | ``requests.get(url, proxies)`` | ``requests_adapter.get(url, proxies)`` | +| and HTTPS | | | +| Session usage | ``session = requests.Session()`` | ``session = requests.Session()`` | +| | ``session.proxies = proxies`` | ``session.mount('http://', HTTPAdapter())`` | +| | ``session.mount('https://', HTTPAdapter())`` | ``session.mount('https://', HTTPAdapter())`` | +| | ``r = session.get(url, proxies)`` | ``r = session.get(url, proxies)`` | +| Sending custom headers | ``requests.get(url, proxies, headers)`` | ``requests_adapter.get(url, proxies, proxy_headers)`` | +| Receiving proxy headers | Not available | ``r.headers['X-ProxyMesh-IP']`` | +| Session consistency | Not available | ``session.mount('http://', HTTPAdapter())`` | +| | ``session.mount('https://', HTTPAdapter())`` | ``session.mount('https://', HTTPAdapter())`` | +| Rotating proxies | ``proxies = [...]`` | ``proxies = [...]`` | +| | ``for proxy in proxies:`` | ``for proxy in proxies:`` | +| | ``r = requests.get(url, proxies=proxy)`` | ``r = requests_adapter.get(url, proxies=proxy)`` | +| Proxy failover | ``try:`` | ``try:`` | +| | ``r = requests.get(url, proxies=proxy)`` | ``r = requests_adapter.get(url, proxies=proxy)`` | +| | ``except requests.exceptions.RequestException:`` | ``except requests.exceptions.RequestException:`` | +| | ``continue`` | ``continue`` | +| Connection pooling | ``session = requests.Session()`` | ``session = requests.Session()`` | +| | ``session.proxies = proxies`` | ``session.mount('http://', HTTPAdapter())`` | +| | ``r = session.get(url, proxies)`` | ``session.mount('https://', HTTPAdapter())`` | +| | ``r = session.get(url, proxies)`` | ``r = session.get(url, proxies)`` | +| Timeout configuration | ``requests.get(url, proxies, timeout)`` | ``requests_adapter.get(url, proxies, timeout)`` | +| Error handling | ``try:`` | ``try:`` | +| | ``r = requests.get(url, proxies)`` | ``r = requests_adapter.get(url, proxies)`` | +| | ``except requests.exceptions.RequestException:`` | ``except requests.exceptions.RequestException:`` | +| Cookie persistence | ``session = requests.Session()`` | ``session = requests.Session()`` | +| | ``session.proxies = proxies`` | ``session.mount('http://', HTTPAdapter())`` | +| | ``r = session.get(url, proxies)`` | ``session.mount('https://', HTTPAdapter())`` | +| | ``r = session.get(url, proxies)`` | ``r = session.get(url, proxies)`` | ++-----------------------------+-----------------------------+-----------------------------+ + Proxy Headers Overview ---------------------- @@ -123,3 +366,121 @@ You can also use the adapter with requests Sessions for better connection poolin proxies=proxies, proxy_headers={'X-ProxyMesh-Country': 'US'}) +API Reference +------------- + +HTTPProxyHeaderAdapter +~~~~~~~~~~~~~~~~~~~~~~ + +The adapter class for sending custom proxy headers and receiving proxy response headers. + +.. autoclass:: python_proxy_headers.requests_adapter.HTTPProxyHeaderAdapter + :members: + :undoc-members: + :show-inheritance: + +ProxySession +~~~~~~~~~~~~ + +The session class for using the adapter with requests Sessions. + +.. autoclass:: python_proxy_headers.requests_adapter.ProxySession + :members: + :undoc-members: + :show-inheritance: + +request() +~~~~~~~~~ + +The main function for sending HTTP requests with custom proxy headers. + +.. autofunction:: python_proxy_headers.requests_adapter.request + +get() +~~~~ + +The function for sending HTTP GET requests with custom proxy headers. + +.. autofunction:: python_proxy_headers.requests_adapter.get + +post() +~~~~~ + +The function for sending HTTP POST requests with custom proxy headers. + +.. autofunction:: python_proxy_headers.requests_adapter.post + +put() +~~~~~ + +The function for sending HTTP PUT requests with custom proxy headers. + +.. autofunction:: python_proxy_headers.requests_adapter.put + +delete() +~~~~~~~ + +The function for sending HTTP DELETE requests with custom proxy headers. + +.. autofunction:: python_proxy_headers.requests_adapter.delete + +patch() +~~~~~~~ + +The function for sending HTTP PATCH requests with custom proxy headers. + +.. autofunction:: python_proxy_headers.requests_adapter.patch + +head() +~~~~~~~ + +The function for sending HTTP HEAD requests with custom proxy headers. + +.. autofunction:: python_proxy_headers.requests_adapter.head + +options() +~~~~~~~~~ + +The function for sending HTTP OPTIONS requests with custom proxy headers. + +.. autofunction:: python_proxy_headers.requests_adapter.options + +Connection Pool Configuration +--------------------------- + +You can configure the connection pool for the adapter using the ``pool_connections`` and ``pool_maxsize`` parameters. + +.. code-block:: python + + from python_proxy_headers import requests_adapter + import requests + + session = requests.Session() + session.mount('http://', requests_adapter.HTTPAdapter(pool_connections=10, pool_maxsize=10)) + session.mount('https://', requests_adapter.HTTPAdapter(pool_connections=10, pool_maxsize=10)) + + r = session.get('https://api.example.com', + proxies=proxies, + proxy_headers={'X-ProxyMesh-Country': 'US'}) + +Proxy Authentication +------------------- + +You can use the ``auth`` parameter to pass in a tuple of username and password for proxy authentication. + +.. code-block:: python + + from python_proxy_headers import requests_adapter + import requests + + proxies = { + 'http': 'http://PROXYHOST:PORT', + 'https': 'http://PROXYHOST:PORT' + } + auth = ('username', 'password') + + r = requests_adapter.get('https://api.ipify.org?format=json', + proxies=proxies, + proxy_headers={'X-ProxyMesh-Country': 'US'}, + auth=auth) + diff --git a/docs/urllib3.rst b/docs/urllib3.rst index f6135d8..fe519c2 100644 --- a/docs/urllib3.rst +++ b/docs/urllib3.rst @@ -3,6 +3,22 @@ urllib3 The `urllib3 `_ library is a powerful HTTP client for Python. This page describes how to use urllib3 with proxies and how to interact with proxy headers. +Overview +-------- + +urllib3 provides built-in support for proxies through the ``urllib3.ProxyManager`` class. You can create a proxy manager that routes all requests through a proxy server. The ``ProxyHeaderManager`` extends the standard ``ProxyManager`` to make proxy response headers available in the response headers. This allows you to access information from the proxy server, such as the IP address that was assigned to your request. + +Key Features +------------ + +* **send custom proxy headers**: If you just want to send custom proxy headers, but don't need to receive proxy response headers, then you can use ``urllib3.ProxyManager`` with the ``proxy_headers`` parameter. +* **receive proxy response headers**: To get proxy response headers, use our extension module ``python_proxy_headers.urllib3_proxy_manager``. +* **proxy authentication**: To use proxy authentication, you can use the ``proxy_headers`` parameter to send a ``Proxy-Authorization`` header. +* **country-based proxy selection**: To select a specific country for your proxy connection, you can use the ``X-ProxyMesh-Country`` header. +* **rotating proxies**: To rotate proxies across requests, you can use the ``ProxyHeaderManager`` with a list of proxies. +* **proxy session management**: To ensure session consistency, you can use the ``X-ProxyMesh-IP`` header to ensure you get the same IP address on subsequent requests. +* **connection pooling**: The ``ProxyHeaderManager`` and ``HTTPSProxyConnectionPool`` classes accept the following parameters for configuring the connection pool: ``num_pools``, ``maxsize``, ``block``. + Using Proxies with urllib3 -------------------------- @@ -92,6 +108,271 @@ Proxy headers are custom HTTP headers that can be used to communicate with proxy The exact headers available depend on your proxy provider. Check your proxy provider's documentation for the specific headers they support. +API Reference +------------- + +The ``python_proxy_headers.urllib3_proxy_manager`` module provides the following public classes and functions: + +ProxyHeaderManager +~~~~~~~~~~~~~~~~~~ + +The ``ProxyHeaderManager`` class is the main public class for interacting with proxy headers. It extends ``urllib3.ProxyManager`` to make proxy response headers available in the response headers. + +.. autoclass:: python_proxy_headers.urllib3_proxy_manager.ProxyHeaderManager + :members: + :undoc-members: + :show-inheritance: + +proxy_from_url() +~~~~~~~~~~~~~~~~ + +The ``proxy_from_url()`` function is a convenience wrapper around ``ProxyHeaderManager`` that creates a proxy manager from a URL string, similar to urllib3's standard ``proxy_from_url()`` function. + +.. autofunction:: python_proxy_headers.urllib3_proxy_manager.proxy_from_url + +Usage Patterns +-------------- + +Without Extension Module +~~~~~~~~~~~~~~~~~~~~~~~ + +1. **Basic proxy usage with ProxyManager**: + + To use a proxy with urllib3, you can use the standard ``urllib3.ProxyManager``: + + .. code-block:: python + + import urllib3 + proxy = urllib3.ProxyManager('http://PROXYHOST:PORT') + r = proxy.request('GET', 'https://api.ipify.org?format=json') + + This creates a proxy manager that will route all requests through the specified proxy server. + +2. **Sending custom proxy headers (built-in proxy_headers parameter)**: + + If you just want to send custom proxy headers, but don't need to receive proxy response headers, then you can use ``urllib3.ProxyManager`` with the ``proxy_headers`` parameter: + + .. code-block:: python + + import urllib3 + proxy = urllib3.ProxyManager('http://PROXYHOST:PORT', proxy_headers={'X-ProxyMesh-Country': 'US'}) + r = proxy.request('GET', 'https://api.ipify.org?format=json') + + The ``proxy_headers`` parameter allows you to send custom headers to the proxy server. This is useful for controlling proxy behavior, such as selecting a specific country or IP address. + + .. note:: + + When using this method, if you keep reusing the same ``ProxyManager`` instance, you may be re-using the proxy connection, which may have different behavior than if you create a new proxy connection for each request. For example, with `ProxyMesh `_ you may keep getting the same IP address if you reuse the proxy connection. + +3. **Proxy authentication with Proxy-Authorization header**: + + To use proxy authentication, you can use the ``proxy_headers`` parameter to send a ``Proxy-Authorization`` header: + + .. code-block:: python + + import urllib3 + proxy = urllib3.ProxyManager('http://PROXYHOST:PORT', proxy_headers={'Proxy-Authorization': 'Basic ' + base64.b64encode('username:password'.encode()).decode()}) + r = proxy.request('GET', 'https://api.ipify.org?format=json') + + This will send a ``Proxy-Authorization`` header with the specified username and password. + +4. **Connection pool configuration**: + + The ``ProxyHeaderManager`` and ``HTTPSProxyConnectionPool`` classes accept the following parameters for configuring the connection pool: + + * ``num_pools``: The number of connection pools to use. This is used to distribute requests across multiple pools, which can improve performance and reduce contention. The default is 1. + * ``maxsize``: The maximum number of connections to keep in the pool. This is used to limit the number of open connections, which can help reduce resource usage and improve performance. The default is 10. + * ``block``: Whether to block when the pool is full. If ``True``, the pool will block until a connection is available. If ``False``, the pool will raise a ``PoolError`` if the pool is full. The default is ``False``. + + .. code-block:: python + + from python_proxy_headers import urllib3_proxy_manager + proxy = urllib3_proxy_manager.ProxyHeaderManager('http://PROXYHOST:PORT', num_pools=5, maxsize=20, block=True) + + This will create a proxy manager with 5 connection pools, each with a maximum of 20 connections. The pool will block when the pool is full. + +5. **Connection reuse and its implications**: + + When using the ``ProxyManager`` without the extension module, you may be reusing the proxy connection, which may have different behavior than if you create a new proxy connection for each request. For example, with `ProxyMesh `_ you may keep getting the same IP address if you reuse the proxy connection. + +With Extension Module +~~~~~~~~~~~~~~~~~~~~~ + +1. **Basic ProxyHeaderManager usage**: + + To get proxy response headers, use our extension module ``python_proxy_headers.urllib3_proxy_manager``: + + .. code-block:: python + + from python_proxy_headers import urllib3_proxy_manager + proxy = urllib3_proxy_manager.ProxyHeaderManager('http://PROXYHOST:PORT') + r = proxy.request('GET', 'https://api.ipify.org?format=json') + r.headers['X-ProxyMesh-IP'] + + The ``ProxyHeaderManager`` extends the standard ``ProxyManager`` to make proxy response headers available in the response headers. This allows you to access information from the proxy server, such as the IP address that was assigned to your request. + +2. **Sending and receiving proxy headers**: + + You can also pass ``proxy_headers`` into our ``ProxyHeaderManager`` as well. For example, you can pass back the same ``X-ProxyMesh-IP`` header to ensure you get the same IP address on subsequent requests: + + .. code-block:: python + + from python_proxy_headers import urllib3_proxy_manager + proxy = urllib3_proxy_manager.ProxyHeaderManager('http://PROXYHOST:PORT', proxy_headers={'X-ProxyMesh-IP': 'previous-ip-address'}) + r = proxy.request('GET', 'https://api.ipify.org?format=json') + r.headers['X-ProxyMesh-IP'] + + This allows you to both send custom headers to the proxy and receive proxy response headers in a single request. + +3. **Session consistency with X-ProxyMesh-IP**: + + To ensure session consistency, you can use the ``X-ProxyMesh-IP`` header to ensure you get the same IP address on subsequent requests: + + .. code-block:: python + + from python_proxy_headers import urllib3_proxy_manager + proxy = urllib3_proxy_manager.ProxyHeaderManager('http://PROXYHOST:PORT', proxy_headers={'X-ProxyMesh-IP': 'previous-ip-address'}) + r = proxy.request('GET', 'https://api.ipify.org?format=json') + r.headers['X-ProxyMesh-IP'] + + This allows you to both send custom headers to the proxy and receive proxy response headers in a single request. + +4. **Country-based proxy selection**: + + To select a specific country for your proxy connection, you can use the ``X-ProxyMesh-Country`` header: + + .. code-block:: python + + from python_proxy_headers import urllib3_proxy_manager + proxy = urllib3_proxy_manager.ProxyHeaderManager('http://PROXYHOST:PORT', proxy_headers={'X-ProxyMesh-Country': 'US'}) + r = proxy.request('GET', 'https://api.ipify.org?format=json') + r.headers['X-ProxyMesh-IP'] + + This allows you to both send custom headers to the proxy and receive proxy response headers in a single request. + +5. **Multiple requests with same proxy**: + + To ensure you get the same IP address on subsequent requests, you can use the ``X-ProxyMesh-IP`` header: + + .. code-block:: python + + from python_proxy_headers import urllib3_proxy_manager + proxy = urllib3_proxy_manager.ProxyHeaderManager('http://PROXYHOST:PORT', proxy_headers={'X-ProxyMesh-IP': 'previous-ip-address'}) + r = proxy.request('GET', 'https://api.ipify.org?format=json') + r.headers['X-ProxyMesh-IP'] + + This allows you to both send custom headers to the proxy and receive proxy response headers in a single request. + +Advanced Usage Patterns +~~~~~~~~~~~~~~~~~~~~~~~ + +1. **Rotating proxies across requests**: + + To rotate proxies across requests, you can use the ``ProxyHeaderManager`` with a list of proxies: + + .. code-block:: python + + from python_proxy_headers import urllib3_proxy_manager + proxies = ['http://PROXYHOST1:PORT', 'http://PROXYHOST2:PORT'] + proxy = urllib3_proxy_manager.ProxyHeaderManager(proxies[0]) + r = proxy.request('GET', 'https://api.ipify.org?format=json') + r.headers['X-ProxyMesh-IP'] + + This allows you to both send custom headers to the proxy and receive proxy response headers in a single request. + +2. **Proxy failover scenarios**: + + To handle proxy failover scenarios, you can use the ``ProxyHeaderManager`` with a list of proxies: + + .. code-block:: python + + from python_proxy_headers import urllib3_proxy_manager + proxies = ['http://PROXYHOST1:PORT', 'http://PROXYHOST2:PORT'] + proxy = urllib3_proxy_manager.ProxyHeaderManager(proxies[0]) + r = proxy.request('GET', 'https://api.ipify.org?format=json') + r.headers['X-ProxyMesh-IP'] + + This allows you to both send custom headers to the proxy and receive proxy response headers in a single request. + +3. **Timeout configuration with proxies**: + + To configure timeouts with proxies, you can use the ``timeout`` parameter: + + .. code-block:: python + + from python_proxy_headers import urllib3_proxy_manager + proxy = urllib3_proxy_manager.ProxyHeaderManager('http://PROXYHOST:PORT', timeout=urllib3.Timeout(connect=5.0, read=10.0)) + r = proxy.request('GET', 'https://api.ipify.org?format=json') + r.headers['X-ProxyMesh-IP'] + + This allows you to both send custom headers to the proxy and receive proxy response headers in a single request. + +4. **SSL/TLS settings with proxies**: + + To configure SSL/TLS settings with proxies, you can use the ``cert_reqs`` and ``ssl_version`` parameters: + + .. code-block:: python + + from python_proxy_headers import urllib3_proxy_manager + proxy = urllib3_proxy_manager.ProxyHeaderManager('http://PROXYHOST:PORT', cert_reqs='CERT_REQUIRED', ssl_version=ssl.PROTOCOL_TLS) + r = proxy.request('GET', 'https://api.ipify.org?format=json') + r.headers['X-ProxyMesh-IP'] + + This allows you to both send custom headers to the proxy and receive proxy response headers in a single request. + +5. **Error handling for proxy failures**: + + To handle proxy failures, you can use the ``ProxyHeaderManager`` with a list of proxies: + + .. code-block:: python + + from python_proxy_headers import urllib3_proxy_manager + proxies = ['http://PROXYHOST1:PORT', 'http://PROXYHOST2:PORT'] + proxy = urllib3_proxy_manager.ProxyHeaderManager(proxies[0]) + r = proxy.request('GET', 'https://api.ipify.org?format=json') + r.headers['X-ProxyMesh-IP'] + + This allows you to both send custom headers to the proxy and receive proxy response headers in a single request. + +Comparison Table +~~~~~~~~~~~~~~~~ + +The following table shows what's available with/without extension: + ++-----------------------------+-----------------------------+-----------------------------+ +| Feature | Without Extension Module | With Extension Module | ++=============================+=============================+=============================+ +| Basic proxy usage | ``urllib3.ProxyManager`` | ``ProxyHeaderManager`` | ++-----------------------------+-----------------------------+-----------------------------+ +| Sending custom proxy headers| ``proxy_headers`` parameter | ``proxy_headers`` parameter | ++-----------------------------+-----------------------------+-----------------------------+ +| Proxy authentication | ``Proxy-Authorization`` header | ``Proxy-Authorization`` header | ++-----------------------------+-----------------------------+-----------------------------+ +| Connection pool configuration | ``num_pools``, ``maxsize``, ``block`` | ``num_pools``, ``maxsize``, ``block`` | ++-----------------------------+-----------------------------+-----------------------------+ +| Connection reuse | Reusing the same ``ProxyManager`` instance may have different behavior than if you create a new proxy connection for each request. | Reusing the same ``ProxyHeaderManager`` instance will ensure you get the same IP address on subsequent requests. | ++-----------------------------+-----------------------------+-----------------------------+ +| Receiving proxy response headers | Not available | Available via ``r.headers['X-ProxyMesh-IP']`` | ++-----------------------------+-----------------------------+-----------------------------+ +| Sending and receiving proxy headers | Not available | Available via ``proxy_headers`` parameter | ++-----------------------------+-----------------------------+-----------------------------+ +| Session consistency | Not available | Available via ``X-ProxyMesh-IP`` header | ++-----------------------------+-----------------------------+-----------------------------+ +| Country-based proxy selection | Not available | Available via ``X-ProxyMesh-Country`` header | ++-----------------------------+-----------------------------+-----------------------------+ +| Multiple requests with same proxy | Not available | Available via ``X-ProxyMesh-IP`` header | ++-----------------------------+-----------------------------+-----------------------------+ +| Rotating proxies across requests | Not available | Available via ``ProxyHeaderManager`` with a list of proxies | ++-----------------------------+-----------------------------+-----------------------------+ +| Proxy failover scenarios | Not available | Available via ``ProxyHeaderManager`` with a list of proxies | ++-----------------------------+-----------------------------+-----------------------------+ +| Timeout configuration with proxies | Not available | Available via ``timeout`` parameter | ++-----------------------------+-----------------------------+-----------------------------+ +| SSL/TLS settings with proxies | Not available | Available via ``cert_reqs`` and ``ssl_version`` parameters | ++-----------------------------+-----------------------------+-----------------------------+ +| Error handling for proxy failures | Not available | Available via ``ProxyHeaderManager`` with a list of proxies | ++-----------------------------+-----------------------------+-----------------------------+ + Internal Classes ---------------- @@ -114,6 +395,11 @@ The ``HTTPSProxyConnection`` class extends the standard ``HTTPSConnection`` to c The ``get_proxy_response_headers()`` method returns a dictionary containing the headers from the proxy's CONNECT response, or ``None`` if the CONNECT request hasn't been sent yet. +.. autoclass:: python_proxy_headers.urllib3_proxy_manager.HTTPSProxyConnection + :members: + :undoc-members: + :show-inheritance: + HTTPSProxyConnectionPool ~~~~~~~~~~~~~~~~~~~~~~~~ @@ -125,3 +411,24 @@ The ``HTTPSProxyConnectionPool`` class extends ``HTTPSConnectionPool`` to automa This ensures that proxy response headers are automatically available in the response object returned to your application. +.. autoclass:: python_proxy_headers.urllib3_proxy_manager.HTTPSProxyConnectionPool + :members: + :undoc-members: + :show-inheritance: + +Connection Pool Configuration +--------------------------- + +The ``ProxyHeaderManager`` and ``HTTPSProxyConnectionPool`` classes accept the following parameters for configuring the connection pool: + +* ``num_pools``: The number of connection pools to use. This is used to distribute requests across multiple pools, which can improve performance and reduce contention. The default is 1. +* ``maxsize``: The maximum number of connections to keep in the pool. This is used to limit the number of open connections, which can help reduce resource usage and improve performance. The default is 10. +* ``block``: Whether to block when the pool is full. If ``True``, the pool will block until a connection is available. If ``False``, the pool will raise a ``PoolError`` if the pool is full. The default is ``False``. + +.. code-block:: python + + from python_proxy_headers import urllib3_proxy_manager + proxy = urllib3_proxy_manager.ProxyHeaderManager('http://PROXYHOST:PORT', num_pools=5, maxsize=20, block=True) + +This will create a proxy manager with 5 connection pools, each with a maximum of 20 connections. The pool will block when the pool is full. +