I've been running a Squid proxy on a dedicated appliance for years, handling all our egress HTTP/HTTPS traffic for the research team. It was solid, predictable. We recently decided to modernize and containerize everything, moving the proxy to a Kubernetes pod with a more current software stack (we chose a popular open-source Layer 7 proxy written in Go for its modern feature set). The goal was better integration with our service mesh and more granular, dynamic policy control.
The migration itself was smooth, but the performance hit has been severe. Where the old hardware box could handle our team's concurrent scanning and fuzzing sessions with sub-100ms additional latency, the new setup introduces 500-2000ms of delay on most requests. This is crippling for interactive tooling. CPU and memory on the new host are barely touched, so it's not a resource starvation issue. I suspect it's a configuration or architectural mismatch.
My current working theory is that we've lost the benefit of dedicated TCP offload and kernel-level tuning that the appliance had. The software proxy is doing everything in userspace. I'm also questioning our TLS inspection setup. Here's a simplified version of our core proxy configuration:
```yaml
forward_proxy:
enabled: true
tls_inspection:
enabled: true
ca_cert: "/certs/internal-ca.pem"
access_log:
enabled: true
format: detailed
buffer_settings:
max_request_bytes: 10485760
max_response_bytes: 10485760
```
We're running it on a node with 8 vCPUs and 32GB RAM, which should be overkill for our ~50 concurrent user load. Network metrics show no packet loss or saturation at the host or pod level.
Has anyone else made a similar transition from a purpose-built hardware middlebox to a software implementation in a virtualized/containerized environment? I'm particularly interested in:
* Specific kernel parameters (`net.core.*`, `net.ipv4.*`) that are critical for high-connection, low-latency proxy workloads in Linux.
* Whether you found significant performance differences between running the proxy as a container vs. a bare-metal process, and any tuning done to mitigate it.
* Experience with TLS decryption/re-encryption overhead in software, and if hardware acceleration (even in VMs) is a hard requirement for acceptable performance.
* Debugging approaches. I've been using `tcpdump` on the proxy and client, and the delays seem to be entirely within the proxy's processing, not network transit.
The move was supposed to increase our agility and control, but right now it's a bottleneck. I'm hoping this is just a matter of missing a few key optimizations rather than a fundamental limitation of the software approach.
hardened by default
Your theory about losing TCP offload and kernel tuning is spot on. That hardware appliance was likely doing TLS and packet reassembly on dedicated silicon, not in a Go runtime's GC cycle.
But I'd bet real money your performance cratered because of a TLS inspection config mismatch. You said "simplified version of" and cut off. If you're terminating and re establishing TLS for inspection on every request inside the container, without a properly sized session cache or connection pooling, you're adding full handshake latency constantly. That's an easy 500-2000ms right there.
Check your proxy's metrics for TLS handshakes vs reused sessions. Also, make sure you're not doing full HTTP/2 stream multiplexing analysis on every packet if you don't need it. Those "modern features" are murder on raw throughput.
Trust me, I'm a pentester.
That's a really good point about TLS handshake overhead, something I wouldn't have considered. It makes me wonder if the default container resource limits could be starving the session cache, too, like if it's not getting enough memory to keep those sessions alive.
How would you even begin to check the TLS metrics for a proxy like that? Is it usually built into the admin interface, or do you need to scrape specific logs?
The memory starvation angle is a red herring. Session caches don't need gigabytes, they need stable storage, which a container's ephemeral memory provides just fine. The real issue is likely that session tickets or cache keys aren't being managed correctly across pod restarts or scaling events.
For checking metrics, you're usually looking at exporter endpoints on the admin port (like /metrics). The specific names are a nightmare - look for `tls_handshakes_total`, `tls_handshake_failures_total`, maybe `tls_session_resumptions`. But counting handshakes doesn't tell you why they're happening.
The more direct test is to run a simple client script that reconnects to the same backend and check if the TLS session is actually being resumed. Most proxies have debug logs that will shout "session resumed" or "full handshake" if you turn the verbosity up. That's your starting point.
If it's not in the threat model, it's not secure.
You're on the right track with the TCP offload, but that's a constant tax, not a 500-2000ms per-request delay. That magnitude points to something synchronous blocking the request path.
Your cut-off point about the TLS inspection setup is the critical lead. If this Go proxy is doing full TLS termination with a 'simplified' configuration, I'd bet it's performing a fresh certificate validation chain for every upstream connection. That means OCSP/CRL checks or even a full path building operation on each request, which is network-bound and can easily hit those latencies. Hardware appliances often have massive, pre-warmed caches for this.
Check if your config has directives like `insecure_skip_verify` or if you've pinned a specific set of CAs. The difference between validating against a local CA store and making external revocation checks is enormous.
~Oli
Yeah, your cutoff point about the TLS inspection setup is exactly where I'd start. Hardware boxes often cheat by having huge CRL/OCSP caches and sometimes even skip online checks by default.
If your new config is doing a full cert validation chain per request, including hitting OCSP responders, you're adding network round trips. That's a huge chunk of that 500-2000ms.
You can test this by temporarily adding an `insecure_skip_verify` equivalent (or disabling OCSP) just to see if the delay vanishes. If it does, you know it's the validation path and not the basic TLS handshake.
Don't trust the model.