System design · Meta Reality Labs · ML platform

An edge gateway for real-time ML, and the ingress controller under it

Getting live sensor streams from AR devices into models on a locked-down GPU cluster meant two builds: a streaming edge gateway that multiplexes thousands of tunnels over one connection, and a from-scratch Kubernetes ingress controller — because the standard one wasn't allowed in the cluster.

role · platform / infra engineerstack · C++ · Python · NGINX · K8s · HTTP/2result · 2× throughput · zero-downtimedomain · real-time ML serving

Context

A contextual-AI system for AR devices needed to feed real-time camera and sensor streams from devices, across a network boundary, into models running on a restricted, GPU-backed cluster. Two things stood in the way: the front-door router was a simple redirector that couldn't stream efficiently across networks, and the cluster's security posture blocked the standard NGINX ingress controller outright. So I built both pieces.

Part 1 — from redirector to edge gateway

The existing gateway just looked up a destination and forwarded a request. Real-time streaming through a cross-network forward proxy needs much more: persistent tunnels, WebSocket support, binary streaming, and flow control. I evolved it into a proper edge gateway with a clean split between a control plane (auth, redirect lookup, routing decisions) and a data plane (fast byte forwarding once the route is established).

Conditional forward-proxy routing — direct-forward vs. proxy-tunnel chosen per resolved destination.
WebSocket tunneling over HTTP CONNECT — and I had to fix the HTTP/1 codec that wasn't performing the upgrade handshake correctly.
Octet-stream forwarding without buffering whole payloads — essential for continuous camera samples.
End-to-end backpressure — a slow downstream throttles the upstream sender safely instead of blowing up memory.
mTLS over the CONNECT tunnel — service-identity certs, so the proxy can authorize who talks to what.

The key move: multiplex, don't reconnect

The expensive part of a secure tunnel is the TLS handshake. Instead of one connection per client, I introduced a persistent HTTP/2 connection cache: many client tunnels ride as independent CONNECT streams inside a single, already-authenticated session.

One handshake instead of a hundred — the throughput win in one picture.

Before — per-request connections

100 clients → 100 TCP handshakes
→ 100 TLS handshakes
→ 100 sockets & file descriptors
latency + CPU scale with clients

After — pooled HTTP/2

1 TCP + 1 TLS handshake, reused
100 CONNECT streams inside it
handshake cost paid once
~2× throughput, far fewer sockets

Part 2 — a Kubernetes ingress controller, from scratch

The cluster wouldn't let me deploy the official NGINX ingress controller (no cluster-wide controller install), but it would run an ordinary Deployment. So I recreated just the functionality needed as a two-container sidecar: unmodified NGINX as the data plane, and a small Python watcher as the control plane. They share an emptyDir volume — the watcher writes config, NGINX reads it. No RPC, no sockets between them.

The same reconcile loop the official controller uses — watch, cache, render, reload.

Design decisions that made it production-grade rather than a script:

Event-driven, not polling — one long-lived watch stream; the controller sleeps until Kubernetes pushes a change.
Desired-state reconciliation — every change regenerates the entire config from cached cluster state, so output is a pure function of current state, not event history. Kills a whole class of drift bugs.
Debounce batching — deploying 40 services triggers one reload, not forty.
Hash-before-reload — a MODIFIED event that doesn't change routing produces no reload.
Atomic writes — tmp → rename() so NGINX never reads a half-written file.
Periodic full resync — watch streams die; a 5-minute relist guarantees eventual consistency.
Zero-downtime reloads — workers finish in-flight requests while new workers pick up the new config.

Deliberate simplification. The official controller watches Services, EndpointSlices, Secrets, IngressClasses and builds upstreams from individual pod IPs. I routed to Kubernetes Services by DNS instead — letting kube-proxy handle discovery and load-balancing — and implemented only the annotation subset the environment actually used (auth, rewrites, proxy timeouts, TLS client verification, snippets). Right-sized, not gold-plated.

Results

2×

cross-network throughput

downtime on config reloads

1 TLS

session for N client tunnels

What I'd do differently

Connection pool, not a single cache. Production edge tiers keep several upstream sessions and pick the least-loaded one; a single cached session is a bottleneck and a failure domain.
Watch EndpointSlices eventually. Routing through Services is simpler, but pod-level upstreams give finer control over draining and retries when you need it.
Ship the WebSocket ping/pong fix upstream. I kept tunnels alive by injecting keepalive frames myself — a pragmatic patch over a codec bug that deserved a real root-cause fix.

Scaling it 10× / 100×

The gateway scales with sessions, not clients, so 10× clients is mostly more streams per session plus horizontal gateway replicas behind a load balancer. At 100× the controller's single watcher becomes the risk: I'd add leader-elected replicas, shard the watch by namespace, and cap reload frequency with a token bucket so a config storm can't melt the data plane. The reconcile model itself — watch, cache, render, hot-reload — holds all the way up; it's the same pattern that runs real clusters.

Written to be public-safe: internal project names, service codenames, colleagues, and partners are generalized to their technical essence. Nothing confidential here — just the engineering.