Add HTTP agent support for connection pooling by paveldudka · Pull Request #619 · apify/proxy-chain

paveldudka · 2025-11-12T18:27:36Z

Problem

Currently, proxy-chain creates a new TCP connection to the upstream proxy for every request. This causes:

New IP addresses per request when using residential/rotating proxies (breaks session-based workflows)
Performance overhead from repeated TCP handshakes
No connection reuse despite HTTP/1.1 keepalive support

Solution

Add optional httpAgent and httpsAgent parameters to prepareRequestFunction result, enabling users to provide persistent HTTP agents with keepAlive: true.

Changes

Add httpAgent? and httpsAgent? to PrepareRequestFunctionResult type
Thread agents through handler pipeline (server.ts, forward.ts, chain.ts)
Pass agents to http.request() and https.request() calls

Benefits

✅ Sticky IP - Same connection = same IP from residential proxy pool
✅ Better performance - Connection reuse eliminates handshake overhead
✅ Backward compatible - Agents are optional, existing code works unchanged
✅ Preserves all existing features - getConnectionStats(), events, etc. still work

Usage

const http = require('http');
const { Server } = require('proxy-chain');

const httpAgent = new http.Agent({
  keepAlive: true,
  keepAliveMsecs: 30000
});

const server = new Server({
  prepareRequestFunction: () => ({
    upstreamProxyUrl: 'http://proxy.example.com:8080',
    httpAgent,  // Reuse connections
  }),
});

Testing

Added comprehensive test suite in test/http-agent.js:

Agent acceptance and threading
Connection pooling verification
Backward compatibility
getConnectionStats() preservation
HTTP vs HTTPS agent selection

All tests passing ✅

Adds optional httpAgent and httpsAgent parameters to prepareRequestFunction result, enabling connection reuse to upstream proxies. Changes: - Add httpAgent/httpsAgent to PrepareRequestFunctionResult type - Add httpAgent/httpsAgent to HandlerOpts type in server, forward, and chain - Pass agents to http.request() and https.request() in forward.ts - Pass agents to upstream CONNECT requests in chain.ts - Thread agents through prepareRequestHandling() in server.ts Benefits: - Enables connection pooling with keepAlive - Supports sticky IP for residential proxies - Backward compatible (agents are optional) - Maintains existing getConnectionStats() functionality 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

jancurn · 2025-11-12T18:43:07Z

Hey @paveldudka this looks cool, thanks for the PR, our team will review it

jirimoravcik · 2025-11-12T19:37:58Z

Hello @paveldudka,
thank you for your contribution!
I ran the tests in CI and noticed your newly added tests are failing. Could you please take a look? Thanks

paveldudka · 2025-11-12T20:26:32Z

Hello @paveldudka, thank you for your contribution! I ran the tests in CI and noticed your newly added tests are failing. Could you please take a look? Thanks

yeah, differences in socket behavior across platforms. Should be fixed now.
Also fixed a test docker container as specified chromium version is not present in deb12 anymore
@jirimoravcik

jirimoravcik · 2025-11-13T15:04:34Z

The tests are still failing, pls @paveldudka

paveldudka · 2025-11-13T15:27:53Z

The tests are still failing, pls @paveldudka

oof. Was fixing linter failure and messed too much with imports, so broke tests...
Now both linter and unit tests should work. At least locally they passed. Could you re-start workflow?
@jirimoravcik @bliuchak

bliuchak

Hi @paveldudka! Big thanks for your contribution and quick fixes 🎉

Here are few issues from my side that should be addressed before merge:

Missing test coverage for CONNECT tunneling (chain handler). Currently all targets in tests are HTTP. Please add a tests for HTTPS target.
Missing HTTPS upstream proxy tests (your last test claims that it verifies HTTPS agent alognside with HTTP, however no code there verifies that httpsAgent is actually used). It would be good to have this covered as separate test.
Document SOCKS limitation. Agents don't work with SOCKS upstream proxies (forward_socks.ts, chain_socks.ts). Would be good to add some JSDoc or mention it in README.
Nice to have: Add usage example of agents into README.

Also, you can treat my comments for code itself as nice to have. Overall implementation looks good. If you'll manage them - good, if not - also good ;)

Let me know if you'll need any clarification.

bliuchak · 2025-11-13T15:05:00Z

+        httpAgent.destroy();
+    });
+
+    it('uses separate agents for HTTP and HTTPS upstream proxies', async () => {


Same as https://github.com/apify/proxy-chain/pull/619/files#r2524036066.

What about instead to verify behavior when using HTTP vs HTTPS upstream proxies (like connection characteristics, actual HTTPS connections being made) rather than checking which internal method was invoked?

bliuchak · 2025-11-13T16:01:11Z

+        httpsAgent.destroy();
+    });
+
+    it('reuses connections with keepAlive agents (sticky IP simulation)', async () => {


What about instead of verifing implementation (hooks into 'free' event, inspect socket implementation details like remoteAddress and port, compare socketIds) to verify an observable effect of connection pooling (behavior)?

What we can do instead is to count actual connections at the upstream proxy server. With this we can verify that fewer connections are made than requests (observable behavior of pooling).

What do you think?

that's a great idea. Lemme dwell on it

paveldudka · 2025-11-13T18:31:48Z

@bliuchak , reworked tests to move away from testing implementation details
Testing https server is quite tricky as I couldn't find a good way to simulate a fully functional https test server. Kinda worked around it by testing that connection is being attempted. LMK what you think

bliuchak · 2025-11-14T12:46:54Z

@paveldudka , thanks for adding examples and document limitation 👏

However the main issue is still there - lack of test coverage for HTTPS upstream proxies (both forward and chain mode).

HTTPS upstream proxy in chain mode.
Current tests cover HTTPS targets through HTTP upstream but missing HTTPS targets through HTTPS upstream.
HTTPS upstream proxy in forward mode.
As you wrote there - current tests uses non-existent proxy URL which is only a smoke test (to verify that code doesn't crash). However what we also need to have is a functional test with real HTTPS upstream proxy and connection pooling verification.

Could you please add these two tests?

paveldudka · 2025-11-15T00:50:59Z

@paveldudka , thanks for adding examples and document limitation 👏

However the main issue is still there - lack of test coverage for HTTPS upstream proxies (both forward and chain mode).

HTTPS upstream proxy in chain mode.
Current tests cover HTTPS targets through HTTP upstream but missing HTTPS targets through HTTPS upstream.

HTTPS upstream proxy in forward mode.
As you wrote there - current tests uses non-existent proxy URL which is only a smoke test (to verify that code doesn't crash). However what we also need to have is a functional test with real HTTPS upstream proxy and connection pooling verification.

Could you please add these two tests?

@bliuchak , If I understand the pipeline correctly, in forward mode (HTTP requests), the proxy terminates each request, allowing the agent to pool TCP connections for subsequent requests. In chain mode (HTTPS via CONNECT), each CONNECT establishes a long-lived tunnel for the client's TLS connection to the target - these tunnels can't be "reused" for different targets, so each requires its own upstream proxy connection.
The httpsAgent still provides value by managing these connections (maxSockets, timeout handling, etc.).
So technically with https upstream we can't really have connection pooling in a traditional sense.

But at this point Im hitting my domain knowledge limitations in this space, so will defer to you to decide how to proceed. Locally I have a forked version of proxy-chain and we use http upstream, so in the short term, we are unblocked

bliuchak · 2025-11-18T09:44:21Z

@paveldudka, I got your point. To be fair we don't have much HTTPS tests with tooling yet (especially for HTTPS upstreams).

What would I propose is to merge this as-is and open a new ticket to add missing coverage later (I can create it).

Here are my arguments:

HTTP upstream part is covered by tests (these upstreams are most used one)
Existing tests are passing
Missing test coverage are mostly localized for HTTPS upstream proxy (forward and chain modes)
To handle traffic from client to HTTPS targets its completely enough to use HTTP upstream and just tunnel traffic using CONNECT
This is relatively a small change

@jirimoravcik What do you think?

jirimoravcik

Thank you, looks great.
One small thing, could we bump the package version in package.json? Thanks

jirimoravcik · 2025-11-18T09:53:23Z

@paveldudka, I got your point. To be fair we don't have much HTTPS tests with tooling yet (especially for HTTPS upstreams).

What would I propose is to merge this as-is and open a new ticket to add missing coverage later (I can create it).

Here are my arguments:

HTTP upstream part is covered by tests (these upstreams are most used one)

Existing tests are passing

Missing test coverage are mostly localized for HTTPS upstream proxy (forward and chain modes)

To handle traffic from client to HTTPS targets its completely enough to use HTTP upstream and just tunnel traffic using CONNECT

This is relatively a small change

@jirimoravcik What do you think?

I agree

bliuchak

@paveldudka please bump package version to 2.6.0 and we're gonna merge it ;)

bliuchak · 2025-11-18T13:26:36Z

FYI #620

paveldudka · 2025-11-18T17:10:23Z

Requested changesMore review options

1 pending review

sounds good. Thanks a lot for a thorough review! @bliuchak

bliuchak · 2025-11-19T09:31:21Z

Released in https://github.com/apify/proxy-chain/releases/tag/v2.6.0

paveldudka and others added 2 commits November 12, 2025 10:21

Add tests

da55ce8

paveldudka changed the title ~~pasha/http agent~~ Add HTTP agent support for connection pooling Nov 12, 2025

paveldudka marked this pull request as ready for review November 12, 2025 18:30

jancurn requested review from bliuchak and jirimoravcik November 12, 2025 18:43

Fix failing unit test

48afd1f

Fix lint errors

088499b

Fix test

758c9e2

bliuchak requested changes Nov 13, 2025

View reviewed changes

paveldudka added 4 commits November 13, 2025 09:44

Address review comments

ba06871

Address review comments

687bc6d

Rework tests

9a5f086

Rework tests

20f18a2

paveldudka requested a review from bliuchak November 13, 2025 18:30

jirimoravcik approved these changes Nov 18, 2025

View reviewed changes

bliuchak requested changes Nov 18, 2025

View reviewed changes

bliuchak mentioned this pull request Nov 18, 2025

Add HTTPS Test Coverage for HTTP Agent Pooling #620

Open

Bump version

d8928e9

paveldudka requested a review from bliuchak November 18, 2025 17:09

bliuchak approved these changes Nov 18, 2025

View reviewed changes

bliuchak merged commit d178688 into apify:master Nov 18, 2025
5 checks passed

Uh oh!

Conversation

paveldudka commented Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Changes

Benefits

Usage

Testing

Uh oh!

jancurn commented Nov 12, 2025

Uh oh!

jirimoravcik commented Nov 12, 2025

Uh oh!

paveldudka commented Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jirimoravcik commented Nov 13, 2025

Uh oh!

paveldudka commented Nov 13, 2025

Uh oh!

bliuchak left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bliuchak Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bliuchak Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

paveldudka Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

paveldudka commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bliuchak commented Nov 14, 2025

Uh oh!

paveldudka commented Nov 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bliuchak commented Nov 18, 2025

Uh oh!

jirimoravcik left a comment

Choose a reason for hiding this comment

Uh oh!

jirimoravcik commented Nov 18, 2025

Uh oh!

bliuchak left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bliuchak commented Nov 18, 2025

Uh oh!

paveldudka commented Nov 18, 2025

Uh oh!

Uh oh!

bliuchak commented Nov 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

paveldudka commented Nov 12, 2025 •

edited

Loading

paveldudka commented Nov 12, 2025 •

edited

Loading

bliuchak left a comment •

edited

Loading

bliuchak Nov 13, 2025 •

edited

Loading

paveldudka commented Nov 13, 2025 •

edited

Loading

paveldudka commented Nov 15, 2025 •

edited

Loading

bliuchak left a comment •

edited

Loading