Confidential LLM Inference over OHTTP

The OpenGradient Python SDK can send OpenAI-compatible chat completions through the OpenGradient Chat API using Oblivious HTTP (OHTTP). The prompt is encrypted for an attested TEE before it leaves the client. The Chat API authenticates the API key and relays ciphertext, but cannot read the prompt or completion.

This path does not require a wallet on the caller. The Chat API relay owns the x402 account used to pay the TEE gateway.

For the protocol and trust model, see Private LLM Inference.

Request Flow

The Python client currently reads the OHTTP configuration and response-signing key from the on-chain TEE Registry. It sends inference traffic to the Chat API relay, not directly to the TEE endpoint.

OpenAI Drop-in Client

OHTTPXClient subclasses httpx.Client, so it can be supplied through the OpenAI Python SDK's existing http_client parameter.

import os

from openai import OpenAI

from opengradient.client.confidential_llm import OHTTPXClient

base_url = "https://chat-api.opengradient.ai"
api_key = os.environ["OPENGRADIENT_API_KEY"]

client = OpenAI(
    api_key=api_key,
    base_url=f"{base_url}/v1",
    http_client=OHTTPXClient(
        relay_url=base_url,
        auth_headers=lambda: {"Authorization": f"Bearer {api_key}"},
    ),
)

response = client.chat.completions.create(
    model="claude-haiku-4-5",
    messages=[{"role": "user", "content": "In one sentence, what is a TEE?"}],
    max_tokens=200,
)

print(response.choices[0].message.content)

The same OpenGradient API key is provided to the OpenAI client and used to authenticate the encrypted OHTTP request. The base_url lets OpenAI construct its normal /chat/completions request. OHTTPXClient intercepts that request and sends the encrypted payload to {base_url}/api/v1/chat/ohttp.

Direct Confidential Client

Use ConfidentialLLM when you do not need OpenAI response objects:

import os

import opengradient as og

base_url = "https://chat-api.opengradient.ai"
api_key = os.environ["OPENGRADIENT_API_KEY"]

client = og.ConfidentialLLM(
    relay_url=base_url,
    auth_headers=lambda: {"Authorization": f"Bearer {api_key}"},
)

result = client.chat(
    model=og.TEE_LLM.CLAUDE_HAIKU_4_5,
    messages=[{"role": "user", "content": "Explain OHTTP in one sentence."}],
    max_tokens=200,
)

print(result.content)
print(f"Verified TEE: {result.proof.tee_id}")
print(f"Signed at: {result.proof.timestamp}")

ConfidentialLLM exposes the verified response and its TeeProof directly. The OpenAI-compatible wrapper performs the same verification before returning an httpx.Response to the OpenAI SDK.

Streaming

Streaming uses chunked OHTTP. The SDK buffers and verifies the encrypted stream before exposing any SSE frames, so the first token is delayed until verification finishes.

stream = client.chat.completions.create(
    model="claude-haiku-4-5",
    messages=[{"role": "user", "content": "Explain confidential inference."}],
    max_tokens=300,
    stream=True,
)

for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

API Key Authentication

Set your OpenGradient API key in the environment before running an example:

export OPENGRADIENT_API_KEY="your-api-key"

Pass that value as OpenAI(api_key=...) and use the same key in the Authorization: Bearer header supplied to OHTTPXClient, as shown in the complete example above.

Debugging the OHTTP Flow

Enable debug=True to inspect the complete local flow:

http_client = OHTTPXClient(
    relay_url=base_url,
    auth_headers=lambda: {"Authorization": f"Bearer {api_key}"},
    debug=True,
)

Debug output includes:

• TEE Registry RPC URL, contract address, selected TEE, and OHTTP key configuration
• Original OpenAI request and the inner request before encryption
• Encrypted request and response bytes in hexadecimal
• Chat API relay URL, response status, and content type
• Decrypted response and successful signature-verification proof
• Encrypted and decrypted chunks for streaming responses

WARNING

Debug mode prints plaintext prompts and completions. Use it only during local development and never enable it in production logs. The API key is not printed.

Current Scope

• OHTTPXClient supports synchronous OpenAI chat-completions requests (POST /chat/completions).
• It does not currently implement AsyncOpenAI or other OpenAI endpoints.
• Streaming is verified before playback rather than delivered token-by-token as it arrives.
• Signature or integrity failures are raised before any decrypted model output is returned.