Node Unblocker Under the Hood: Architecture, Middleware, and Scaling

By Nicholas St. Germain — May 5, 2026

Beyond the Basics: How Node Unblocker Actually Works

Most guides on Node Unblocker show you how to install it and proxy your first request - that ground is covered in our practical Node Unblocker guide, and you should start there if you have not stood up a working server yet. This post is for the next step: when you are planning to run it in production or build something serious on top of it, and you need to understand what is happening beneath the surface.

It covers the internals - the streaming pipeline, URL rewriting engine, cookie transfer mechanism, and client-side injection system - along with practical strategies for scaling and hardening your deployment.

The Streaming Architecture

The defining design decision in Node Unblocker is that nothing is buffered. Unlike older web proxies (CGIProxy, PHProxy, Glype) that download an entire page into memory before parsing and forwarding it, Node Unblocker processes data on the fly using Node.js Transform streams.

Here's what that looks like in practice:

Client Request → Express Middleware → Remote Request
                                          ↓
Client ← Transform Stream Pipeline ← Remote Response

Each piece of middleware that needs to modify the response body creates a Transform stream, and these streams are piped together in sequence. The data flows through this pipeline chunk by chunk - the client starts receiving bytes before the remote server has finished sending them.

This matters for two reasons:

1. Memory usage stays flat regardless of response size. You can proxy a 500MB file without allocating 500MB of RAM.
2. Time-to-first-byte is minimized. The client doesn't wait for the full response to be downloaded and processed before seeing any content.

Nathan Friedly, the author, reported serving over 1,200 requests per minute on a single Heroku dyno using this architecture. That's the kind of throughput you get when your proxy isn't fighting the garbage collector.

How URL Rewriting Works

URL rewriting is the most complex part of any web proxy. Node Unblocker takes a "pretty URL" approach with a configurable prefix (default: /proxy/). A request to:

http://your-server.com/proxy/https://example.com/page

gets proxied to https://example.com/page, and all URLs in the response are rewritten to go back through the proxy.

What gets rewritten server-side

The url-prefixer middleware handles two content types:

HTML: All href, src, action, srcset, and similar attributes are rewritten. Absolute URLs get the proxy prefix prepended. The proxy also injects a <meta name="ROBOTS" content="NOINDEX, NOFOLLOW" /> tag to prevent search engines from indexing proxied content.

CSS: URLs inside url() declarations and @import statements are rewritten.

What gets rewritten client-side

JavaScript is too complex and varied to reliably rewrite on the server. Instead, Node Unblocker injects client-side scripts that wrap browser APIs:

• fetch() - intercepted so all fetch calls route through the proxy (added in v2.3.0)
• XMLHttpRequest - the open() method is wrapped to rewrite URLs
• WebSocket - the constructor is wrapped to route connections through the proxy's upgrade handler
• history.pushState / replaceState - wrapped to keep the browser's address bar showing proxy URLs (added in v2.3.0)

This two-layer approach (server-side for static content, client-side for dynamic APIs) is what lets Node Unblocker handle modern single-page applications reasonably well.

Handling relative URLs

Root-relative links (e.g., /images/logo.png) present a challenge because they resolve against the proxy's origin, not the target site. Node Unblocker handles these by checking the Referer header and issuing a 307 redirect to the correct proxied URL. Where possible, it rewrites these links proactively in HTML to avoid the redirect overhead.

The Cookie Transfer Mechanism

Cookie handling in a web proxy is deceptively hard. Cookies are scoped by domain, path, and protocol - all of which change when you're proxying.

Node Unblocker's cookies middleware rewrites the Path attribute on Set-Cookie headers so cookies are scoped to the proxied path. For example, a cookie from example.com with Path=/ gets rewritten to Path=/proxy/http://example.com/.

The harder problem is cross-domain cookie transfer. When a user navigates from http://example.com to https://example.com (protocol switch) or from www.example.com to api.example.com (subdomain switch), cookies need to follow. Node Unblocker handles this with an elegant redirect-based approach:

1. Links that would cross protocol or subdomain boundaries are rewritten to first hit a cookie-handling endpoint
2. That endpoint copies the relevant cookies to the new proxy path
3. Then it issues a redirect to the actual target

The Secure flag is also stripped from cookies, since the proxy might be serving over HTTP even if the original site uses HTTPS.

The Middleware System in Depth

Almost everything Node Unblocker does is implemented as middleware. Understanding this system is key to extending it effectively.

Request middleware

Request middleware functions receive a data object with:

{
  url: 'http://example.com/',       // target URL
  clientRequest: req,                // incoming Express request
  clientResponse: res,               // outgoing Express response
  headers: { /* ... */ },            // headers that will be sent to remote
  stream: ReadableStream             // request body
}

If any middleware sends a response on clientResponse, the pipeline short-circuits - no further middleware runs and no remote request is made. This is how you implement access controls, caching, or request blocking.

Response middleware

Response middleware gets the same object plus remote request/response details:

{
  // ...everything from request data, plus:
  remoteRequest: remoteReq,
  remoteResponse: remoteRes,
  contentType: 'text/html',
  headers: { /* response headers */ },
  stream: ReadableStream              // response body
}

To modify response content, you create a Transform stream and replace data.stream:

const { Transform } = require('stream');

function injectBanner(data) {
  if (data.contentType !== 'text/html') return;

  const transform = new Transform({
    transform(chunk, encoding, callback) {
      const html = chunk.toString();
      const modified = html.replace(
        '</body>',
        '<div style="position:fixed;bottom:0;width:100%;background:#333;' +
        'color:#fff;padding:8px;text-align:center;z-index:99999">' +
        'Viewing through proxy</div></body>'
      );
      callback(null, modified);
    }
  });

  data.stream = data.stream.pipe(transform);
}

Disabling built-in middleware

Setting standardMiddleware: false strips out all built-in behavior, letting you cherry-pick and reorder what you need:

const Unblocker = require('unblocker');

const unblocker = new Unblocker({
  prefix: '/proxy/',
  standardMiddleware: false,
  requestMiddleware: [
    Unblocker.host,          // fix Host header
    Unblocker.referer,       // rewrite Referer header
    myCustomAuthMiddleware,  // your access control
    myCustomCacheMiddleware, // your caching layer
  ],
  responseMiddleware: [
    Unblocker.decompress,    // handle gzip/deflate
    Unblocker.charsets,      // normalize to UTF-8
    Unblocker.urlPrefixer,   // rewrite URLs
    Unblocker.cookies,       // fix cookie paths
    Unblocker.hsts,          // strip HSTS headers
    Unblocker.csp,           // strip CSP headers
    myCustomLogger,          // your logging
  ]
});

The middleware debugger

Node Unblocker ships with a built-in debugging tool that wraps every middleware with before/after logging:

DEBUG=unblocker:middleware node app.js

This shows exactly what each middleware changed in the request or response. It is extremely useful when building custom middleware and trying to understand why a particular site doesn't work correctly through the proxy.

Security Headers and What Gets Stripped

Node Unblocker removes several security headers that would break proxied content:

This is necessary for the proxy to function, but it means proxied content has weaker security protections than the original site. Keep this in mind when deciding what to proxy and who has access.

What Node Unblocker Can't Do

Understanding the limitations is as important as understanding the features:

No OAuth or social login. Sites using OAuth flows (Sign in with Google, Facebook, etc.) or postMessage() will break. The redirect-based auth flow doesn't survive URL rewriting.

Complex SPAs often fail. Discord, Twitter/X, YouTube, Instagram, and similar heavily-scripted sites use techniques that break through the proxy's client-side wrapping.

No Cloudflare bypass. Cloudflare's bot detection, challenge pages, and JavaScript challenges will block proxied requests.

No built-in IP rotation. Node Unblocker uses the IP of the server it runs on. For IP diversity, you need to pair it with an external proxy service or deploy multiple instances.

Only specific content types are processed. By default, only text/html, application/xml+xhtml, application/xhtml+xml, and text/css are modified. Everything else (images, fonts, JavaScript files, JSON responses) passes through unmodified.

AGPL-3.0 license. This is a copyleft license that requires you to release the source code of any application that uses Node Unblocker over a network. Many enterprises (Google explicitly bans AGPL) cannot use it without purchasing a commercial license from the author.

Scaling Strategies

Multi-instance deployment

Since Node Unblocker is stateless (no Redis dependency since v1.0), you can run multiple instances behind a load balancer:

                    ┌─ Unblocker Instance 1
Load Balancer ──────┼─ Unblocker Instance 2
                    └─ Unblocker Instance 3

Each instance runs independently. No shared state means no synchronization overhead.

Pairing with upstream proxies

For IP diversity, route Node Unblocker's outbound requests through different upstream proxies per request:

const HttpsProxyAgent = require('https-proxy-agent');

const proxyPool = [
  'http://user:pass@proxy1.example.com:3128',
  'http://user:pass@proxy2.example.com:3128',
  'http://user:pass@proxy3.example.com:3128',
];

let proxyIndex = 0;

const unblocker = new Unblocker({
  prefix: '/proxy/',
  requestMiddleware: [
    function rotateProxy(data) {
      const proxy = proxyPool[proxyIndex % proxyPool.length];
      data.agent = new HttpsProxyAgent(proxy);
      proxyIndex++;
    }
  ]
});

This gives you the URL rewriting and cookie handling of Node Unblocker with the IP diversity of a proxy pool - combining the best of both.

Connection pooling

For high-throughput deployments, configure custom HTTP agents with connection pooling:

const http = require('http');
const https = require('https');

const unblocker = new Unblocker({
  prefix: '/proxy/',
  httpAgent: new http.Agent({
    keepAlive: true,
    maxSockets: 256,
    maxFreeSockets: 64,
  }),
  httpsAgent: new https.Agent({
    keepAlive: true,
    maxSockets: 256,
    maxFreeSockets: 64,
  }),
});

Reusing TCP connections avoids the overhead of TLS handshakes and TCP slow-start on every request.

Production hardening

A few things to lock down before exposing Node Unblocker to traffic:

const unblocker = new Unblocker({
  prefix: '/proxy/',
  requestMiddleware: [
    // Block access to internal networks
    function blockInternal(data) {
      const url = new URL(data.url);
      const hostname = url.hostname;
      if (
        hostname === 'localhost' ||
        hostname.startsWith('127.') ||
        hostname.startsWith('10.') ||
        hostname.startsWith('192.168.') ||
        hostname.match(/^172\.(1[6-9]|2[0-9]|3[0-1])\./)
      ) {
        data.clientResponse.status(403).send('Access denied');
      }
    },
    // Rate limiting per IP (pseudocode - use express-rate-limit in practice)
    function rateLimit(data) {
      // implement rate limiting logic
    }
  ]
});

Without access controls, an open proxy can be exploited for abuse, port scanning internal networks, or laundering malicious traffic through your server.

Observability: Knowing When the Proxy Is Failing

The streaming architecture that makes Node Unblocker fast also makes it harder to debug. By the time you notice a target site has changed and is now breaking, you may have already streamed thousands of malformed responses to clients. A small amount of structured logging goes a long way.

The pattern I reach for is a response middleware that records the URL, status, content type, and response size for every proxied request. Because middleware runs before the body is streamed, recording the size requires a passthrough Transform that counts bytes and emits the log line on end:

const { Transform } = require('stream');

function metricsMiddleware(data) {
  let bytes = 0;
  const counter = new Transform({
    transform(chunk, _enc, cb) {
      bytes += chunk.length;
      cb(null, chunk);
    },
    flush(cb) {
      console.log(JSON.stringify({
        ts: Date.now(),
        url: data.url,
        status: data.remoteResponse.statusCode,
        contentType: data.contentType,
        bytes,
      }));
      cb();
    },
  });
  data.stream = data.stream.pipe(counter);
}

A few things this gives you cheaply:

• Block-rate alerts. A sudden spike in 403s or 429s for a target hostname almost always means the site rolled out new bot detection.
• Empty-response detection. Target pages that suddenly start responding with 0 bytes (or a fraction of normal size) usually indicate a soft block or an A/B test of an interstitial.
• Per-target latency budgets. Combine timestamp deltas with the URL to spot upstream proxies degrading before they fully fail.

For anything beyond stdout, ship the JSON lines into your existing log aggregator. The Node Unblocker process itself should stay focused - the moment you start doing synchronous I/O inside the middleware pipeline, the streaming benefit evaporates.

Node Unblocker vs. Other Approaches

The right choice depends on your use case. Node Unblocker excels when you need to transform web content and route it through a proxy with full URL rewriting. If you just need to forward API requests, http-proxy-middleware is simpler. If you need to scrape at scale against sites with anti-bot protection, a managed service will save you time.

Pairing Node Unblocker with ISP Proxies

Node Unblocker handles the hard part of content transformation - URL rewriting, cookie management, header stripping - but it doesn't solve the IP reputation problem. Requests still come from your server's IP, which target sites can fingerprint and block.

The most effective approach is to pair Node Unblocker with residential or ISP proxies. ISP proxies give you IP addresses assigned by real internet service providers, making your traffic indistinguishable from a regular user. When you route Node Unblocker's outbound requests through ISP proxies, you get:

• Clean URL rewriting from Node Unblocker
• Trusted IP addresses from the ISP proxy pool
• Lower block rates since ISP IPs don't appear on datacenter blocklists

This combination is particularly effective for monitoring competitor pricing, aggregating public data, or accessing content that's restricted to specific regions - all through a single, self-hosted proxy layer that you fully control.

Wrapping Up

Node Unblocker is more than a quick proxy setup. Its streaming architecture, extensible middleware system, and thoughtful handling of cookies and URLs make it a solid foundation for building custom proxy infrastructure. Understanding these internals helps you debug issues, extend functionality, and make informed decisions about when it's the right tool for the job - and when it isn't.