Private LLM Inference

OpenGradient provides private LLM inference infrastructure that combines Oblivious HTTP (OHTTP) with hardware-attested Trusted Execution Environments (TEEs). Prompts and completions are end-to-end encrypted to an attested enclave, while a network relay separates the client's network identity from the request content.

This is offered as a piece of infrastructure on top of the standard Verifiable LLM Execution stack - the same TEE registry, the same on-chain attestation, and same payments on Base - with an added private inference layer that decouples client identity from request content. It is exposed through the Python SDK and TypeScript SDK so applications can opt in to private inference without managing HPKE or relay routing themselves.

TIP

If you only need verifiable inference (provable prompt usage, signed responses) without identity unlinkability and additional privacy guarantees, see Verifiable LLM Execution. Private inference layers an additional privacy guarantee on top of that.

Key Features

• End-to-end encryption to the enclave - Prompts and completions are sealed under HPKE (RFC 9180) on the client side, using a public key that is bound to an attested enclave build. Only the secure enclave can decrypt in order to forward it to the model provider.
• Identity / content unlinkability - A two-hop architecture (relay + gateway) splits the request so the relay sees the client's IP but not the plaintext, and the enclave sees the plaintext but not the client's IP.
• Registry-anchored key distribution - The client selects an active TEE from the on-chain registry and reads its registered HPKE public key, response-signing key, endpoint, and TEE identity. Attestation validation happens when the TEE is admitted to the registry.
• Streaming support - Token-by-token responses use Chunked OHTTP, so encrypted chunks remain hidden from the relay. The Python SDK buffers and verifies the stream before returning its decrypted SSE frames.
• Signed responses - The enclave signs responses using RSA-PSS-SHA256 over keccak256(requestHash || outputHash || timestamp). The client verifies the signed response before returning it.
• Same model coverage - All models supported by the verifiable LLM stack (OpenAI, Anthropic, Google, xAI, etc.) are reachable through the private endpoint.

Trust Model

Private inference splits trust between two independent network entities and an enclave:

Party	Sees	Does Not See
Client	Plaintext, registry TEE configuration, response proof metadata	-
Relay	Client IP, OHTTP ciphertext, selected TEE	Prompt and completion content
TEE Gateway (enclave)	Request and response, relay IP	Client IP, client identity
Upstream model provider	Request from the enclave's egress	Client IP, client identity, OpenGradient routing

The privacy guarantee is non-collusion between relay and gateway: as long as those two operators do not share data, no party can link a given client to a given prompt. The registry's attestation checks ensure the gateway operator cannot read plaintext outside the approved code path even if they wanted to - the HPKE private key never leaves the enclave's memory, and the verifiable and open-source server code ensures that nothing is logged or recorded.

Architecture Overview

The diagram shows the two-hop split: the relay sees who you are (IP) and the selected TEE, but not the prompt or completion content; the enclave sees the content but not the client's IP. The on-chain TEE Registry anchors trust by publishing active TEE identities and their encryption and signing keys. The client encrypts to the selected registry key and verifies signed responses against the registered signing key.

How It Works

1. Enclave Startup & Key Generation

When a gateway enclave boots, it generates two keypairs inside the TEE:

• An RSA-2048 signing keypair, used to sign inference responses.
• An X25519 HPKE keypair, used as the OHTTP key configuration for encrypting client requests.

Both public keys are bound to a single AWS Nitro attestation document via the nitriding daemon's transcript. Specifically, the attestation's user_data field commits to a transcript of the form:

og-tee-keys|v2|rsa-spki=<DER>|hpke-x25519=<32 bytes>

This binds both keys to the same attested enclave - they cannot be substituted independently.

The enclave is registered on the on-chain TEE Registry on the OpenGradient network, which checks the attestation against the AWS Nitro root CA and confirms the enclave's PCR measurements match an approved build.

2. Key Configuration Distribution

The client selects an active OHTTP-capable LLM TEE from the on-chain TEE Registry and uses its registered key configuration for encryption and response verification.

The selected registry record provides the TEE identity and endpoint, its RSA response-signing public key, and its OHTTP configuration:

{
  "key_id": 1,
  "kem_id": 32,
  "kdf_id": 1,
  "aead_id": 3,
  "public_key": "<hex X25519 public key>",
  "key_config": "<registered OHTTP key config>"
}

The registered ciphersuite is X25519/HKDF-SHA256/ChaCha20-Poly1305. The registry anchors the registered keys and approved PCR measurement on-chain.

3. Encapsulating a Request

The client constructs the inner LLM request - model, messages, parameters - exactly as it would for an OpenAI-compatible API:

{
  "model": "openai/gpt-5",
  "messages": [{"role": "user", "content": "Summarize this contract..."}],
  "temperature": 0.2,
  "stream": false
}

It serializes the inner request as UTF-8 JSON, HPKE-seals those bytes under the enclave's registered X25519 public key, and prefixes the ciphertext with a key-config header.

wire        = header || enc || ciphertext
header      = key_config_id(1B) || kem_id(2B) || kdf_id(2B) || aead_id(2B)
enc         = ephemeral X25519 public key (32B)
ciphertext  = ChaCha20-Poly1305( UTF-8 JSON request )

The client POSTs this opaque blob to the relay:

POST /api/v1/chat/ohttp HTTP/1.1
Host: <relay>
Content-Type: message/ohttp-req

<binary OHTTP request>

The relay cannot read the encrypted inner request. It sees only the sealed bytes, source IP, and selected TEE ID.

4. Relay Forwarding & Payment

The relay attaches an X-Payment header containing a signed x402 payment authorization for $OPG on Base, then forwards the sealed payload to the gateway's /v1/ohttp endpoint. The relay pays the gateway; the relay separately bills its own users on a subscription or per-call basis.

This indirection - relay-paid rather than client-paid at the enclave boundary - is what allows the gateway to charge for inference without ever learning who the end user is. The relay handles payments from clients independently, for example using a subscription model, keeping the user identity separated.

5. Gateway Decryption and Inference

Inside the enclave, the gateway:

1. Verifies the relay's X-Payment header against the x402 facilitator on Base.
2. HPKE-decrypts the OHTTP payload using the private key that never leaves enclave memory.
3. Decodes the inner JSON request and dispatches it to the appropriate upstream provider (OpenAI, Anthropic, Google, xAI, ByteDance ModelArk, etc.) using the enclave's own egress identity.
4. Collects the response, signs it inside the enclave, and seals the signed response back to the client.

The signature uses the RSA-2048 signing key that was bound to the attestation:

msg_hash = keccak256( abi.encodePacked(requestHash, outputHash, timestamp) )
sig      = RSA-PSS-SHA256(signingKey, msg_hash, salt_len=32)

The sealed response includes tee_signature, tee_request_hash, tee_output_hash, tee_timestamp, and tee_id (keccak256 of the signing public key DER) so the client can independently verify the response against the on-chain TEE registry entry.

6. Streaming Responses

For stream: true requests, the gateway returns a Chunked OHTTP response using the message/ohttp-chunked-res MIME type. The wire format is:

response_nonce
( varint(sealed_len) || sealed_chunk_ct )+
varint(0)
sealed_final_ct        // AAD = "final"

Each response chunk is individually AEAD-sealed, so the relay forwards opaque frames without seeing token content. The final chunk uses the AAD "final", so a truncated stream is detected. The Python SDK buffers the stream, decrypts its SSE frames, and verifies the final response signature before returning them.

7. Client Verification

After decrypting an OHTTP response, the client:

1. Recomputes the canonical request hash and compares it with tee_request_hash.
2. Recomputes the output hash and compares it with tee_output_hash.
3. Confirms that tee_id matches the selected registry entry.
4. Verifies the RSA-PSS signature against the registered signing key.

Missing or invalid proof metadata causes the SDK to reject the response. Successful verification proves that:

• The response came from an enclave running approved code (TEE Registry + attestation).
• The exact prompt the client encrypted is the prompt that was answered (request hash).
• The exact response the client received is the response the enclave produced (output hash + signature).
• The relay could not have read the prompt or completion (HPKE confidentiality).
• The gateway could not have learned the client's IP (relay indirection).

Attestation Lifecycle

Phase	What Happens
Build	Enclave image is built reproducibly. PCR0/PCR1/PCR2 measurements are recorded.
Approval	PCR hashes are added to the on-chain approved list via the TEE Registry's admin process.
Boot	Enclave generates RSA + X25519 keypairs inside TEE memory. Keys are committed to nitriding's attestation transcript.
Registration	Enclave registers itself with the on-chain TEE Registry, providing the attestation, signing key, TLS cert, payment address, and endpoint. The contract verifies the attestation, PCRs, and key bindings.
Serving	Clients select an active TEE and read its OHTTP and signing keys from the on-chain registry before sending sealed requests.
Key rotation	A new keypair triggers a new attestation, registration, and key config; clients use it on their next registry lookup.

See Verifiable LLM Execution → TEE Registry for the full set of on-chain checks performed during registration.

What Is and Isn't Hidden

Visible to relay	Visible to gateway	Visible to upstream model provider
Client IP	Relay IP	Enclave egress IP
Sealed OHTTP request bytes, selected TEE	Plaintext prompt	Plaintext prompt
OHTTP response bytes	Plaintext completion	Plaintext completion
x402 cost settlement (non-stream)	Token usage, cost	Token usage
Request timing	Model selected	Model selected

IMPORTANT

Private inference protects content and identity unlinkability, not metadata about traffic timing or volume. A network observer that sees both the client's connection to the relay and the relay's connection to the gateway can still perform traffic-analysis correlation. Sensitive deployments should consider running the relay and the gateway under independent operators and on independent networks.

Comparison with Verifiable LLM Execution

Property	Verifiable LLM Execution	Private LLM Inference
Hardware-attested execution	✅	✅
Signed response proof	✅	✅ when signed metadata is present
Prompt confidential from TEE operator	✅ (TLS to enclave)	✅ (HPKE to enclave)
Prompt confidential from network observer	✅ (TLS)	✅ (HPKE)
Client IP hidden from inference enclave	❌	✅
Client identity decoupled from payment	❌	✅
Transport	HTTPS + x402	OHTTP + HPKE + x402
Wire MIME type	application/json	message/ohttp-req / message/ohttp-res / message/ohttp-chunked-res

Standards & References

• RFC 9458 - Oblivious HTTP
• RFC 9180 - Hybrid Public Key Encryption (HPKE)
• draft-ietf-ohai-chunked-ohttp - Chunked OHTTP
• x402 Payment Standard
• AWS Nitro Enclaves Attestation

Next Steps

• Read Verifiable LLM Execution for the underlying TEE registration, attestation, and settlement mechanics that private inference inherits.
• See Proof Settlement for how inference proofs are recorded on the OpenGradient network.
• Use Confidential LLM Inference over OHTTP for the Python SDK and OpenAI-compatible examples.