GPUs Don't Have Firewalls: Preventing Multi-Tenant Context Leaks in AI Datacenters

Technical level: Intermediate to advanced. This article is intended for security architects, AI engineers, platform engineers, cloud architects and CISOs responsible for designing and securing multi-tenant AI infrastructure.

In traditional enterprise cloud architecture, we’ve spent the last fifteen years perfecting the art of isolation. We segregate workloads using Virtual Private Clouds (VPCs), configure security groups, enforce network micro-segmentation, and let OS-level hypervisors cleanly slice CPU and system memory among tenants.

This model works because standard CPU registers and RAM architectures have decades of hardened, hardware-level isolation layers built into them.

But the moment you step into a modern AI datacenter, this entire security paradigm falls apart.

To run high-throughput LLM inference, RAG pipelines, or real-time model orchestration, we rely on high-performance physical GPU pools, such as NVIDIA H100, H200, or the newer Blackwell architectures. Because these cards are extremely expensive, nobody assigns a dedicated, physical $30,000 GPU to a single lightweight service.

Instead, we virtualise them. We use technologies like NVIDIA’s Multi-Instance GPU (MIG) or virtual GPUs (vGPU) to slice a physical card into multiple virtual fractions, sharing the same physical silicon and unified VRAM among completely different containerized applications, models, or even external tenants.

And that is where the nightmare begins.

GPUs are engineered for raw, uncompromising mathematical throughput and massive parallel bandwidth. They are not general-purpose CPUs. They do not have Layer-7 firewalls built into the silicon.

If a kernel-level exploit, a zero-day vulnerability in the orchestration layer (like Kubernetes or Triton Inference Server), or a simple memory-management bug occurs, there is nothing at the hardware layer to prevent physical "VRAM bleed." Your highly sensitive prompt payloads, system instructions, and vector embeddings can leak directly into the memory register of another tenant sharing that same card.

The Blindspot: Where Network Security Meets Silicon

When most security architects audit an AI pipeline, they look at the network diagram. They verify that the containerised RAG service is running inside a secure, private Kubernetes subnet, and that Triton or vLLM is behind an HTTPS gateway.

They tick the compliance box and walk away, completely blind to the fact that once the data crosses the API gateway, it is loaded as raw plaintext into a shared physical VRAM pool.

Consider this standard multi-tenant inference flow:

If Tenant B executes a malicious prompt injection or triggers a kernel-level memory boundary overrun on the shared NVRM (NVIDIA Kernel Driver) layer, they can read adjacent VRAM buffers. If your financial application just sent a prompt containing a customer's credit score, transaction history, or KYC details to that same card, that data is now sitting in raw registers waiting to be scraped.

The network firewall can protect the packet while it's in transit. The database encryption can protect the packet while it's at rest. But the moment that data is in-use inside the GPU, it is completely exposed.

The Verdict: Zero-Code Proxy-Level Isolation with APIgator

If you are a Chief Information Security Officer (CISO) or a Lead Architect in a highly regulated sector like Banking or Insurance, you cannot afford to assume that physical hardware slicing holds under a zero-day exploit. To satisfy the strict data-protection mandates of the EU AI Act, PCI-DSS v4.0, and GDPR, you must treat the entire physical GPU memory space as untrusted.

The only pragmatic, peer-reviewed solution is to establish a Zero-Trust Model-Ingestion Gateway.

But here is where many development teams get it wrong: they try to compile proprietary cryptography libraries or complex, custom SDKs directly into their microservices. This creates heavy dependencies, slows down development velocity, and degrades model performance.

eXate’s design philosophy bypasses this overhead completely through APIgator.

APIgator is deployed as an inline proxy or sidecar gateway directly in front of your model-server inference endpoints (like Triton or vLLM). Because APIgator gates at the proxy level, no code modifications are required in your application.

The application simply routes its inference requests to the APIgator endpoint instead of the raw Triton/vLLM port.

APIgator intercepts the raw HTTP POST request containing the JSON prompt payload, processes the text in microseconds using policies compiled out-of-band by GatorAid, and automatically swaps sensitive attributes (such as account numbers, financial figures, and PII) with secure, reversible synthetic tokens before the payload is loaded into the shared GPU memory.

• Prompt Sent by App: "Summarize the risk profile of account 9876-54321, showing a ledger deficit of $120,500."

• APIgator Sanitized Payload Sent to GPU: "Summarize the risk profile of account [Ab.b4.Hx-bvcsdjkbv djsb], showing a ledger deficit of [db.Dh.j8-dsalfjbhdsjkgvb]."

  
  

Even if a VRAM bleed or container breakout occurs inside the shared GPU, the only thing leaked is a series of deterministic, useless ciphers. When the model returns the generated completion, APIgator intercepts the response, evaluates the caller's authorized attributes using Attribute-Based Access Control (ABAC), and dynamically decrypts the sensitive values back into plaintext safely outside the untrusted GPU environment.

The Sovereign Takeaway

The ultimate lesson of the modern AI datacenter is simple: you cannot outsource trust to the hardware layer.

Whether you are using specialized GPU-as-a-Service cloud providers or building out private, multi-tenant Kubernetes clusters, physical resource sharing is an economic necessity. But sharing physical resources means sharing physical risks.

Sovereignty is not a physical boundary; it is a runtime control loop.

Stop assuming that network-level VPCs or hypervisors protect your in-memory assets. Start assuming that every memory register on your GPU is public, and design your ingestion pipelines accordingly.

Because if your prompts aren't tokenised before they hit the GPU, your multi-tenant cluster isn't just running model inference. It is running a multi-tenant context leak.