Skip to main content

​
Service tiers

Terminal provides two inference windows.
WindowBest forExpected behavior
StandardInteractive requests, agents, user-facing chatFast responses, 100+ tokens/sec on average
FlexOffline jobs, batch-like work, cost-sensitive workloadsSame endpoint waits for completion, usually 5-20 minutes

​
Standard window

Standard is the default window. Use it when latency matters or when the model is serving an interactive workflow. 
from openai import OpenAI

client = OpenAI(
    base_url="https://api.terminal.fyi/v1",
    api_key="TERMINAL_API_KEY",
)

response = client.chat.completions.create(
    model="gemma-4-31b",
    messages=[
        {
            "role": "user",
            "content": "Turn this deploy log into a one-paragraph on-call update.",
        }
    ],
    max_tokens=8000,
)

print(response.choices[0].message.content)

​
Flex window

Flex is for workloads that can wait. It uses the same Chat Completions endpoint and returns when the request completes. Configure your HTTP client with a long timeout. Gemma 4 31B supports the Flex window with service_tier: "flex". 
from openai import OpenAI

client = OpenAI(
    base_url="https://api.terminal.fyi/v1",
    api_key="TERMINAL_API_KEY",
    timeout=1800.0,
)

response = client.chat.completions.create(
    model="gemma-4-31b",
    service_tier="flex",
    messages=[
        {
            "role": "user",
            "content": "Review these nightly eval notes and return three follow-up tasks.",
        }
    ],
    max_tokens=8000,
)

print(response.choices[0].message.content)
Use Flex for background jobs, long-running analysis, and other work where a 5-20 minute response time is acceptable.

​
Custom SLAs

Terminal will support request-level SLA hints in headers so you can define latency and cost preferences without changing the request body. These headers are not live yet. 
from openai import OpenAI

client = OpenAI(
    base_url="https://api.terminal.fyi/v1",
    api_key="TERMINAL_API_KEY",
)

response = client.chat.completions.create(
    model="gemma-4-31b",
    messages=[
        {
            "role": "user",
            "content": "Extract launch-review action items and group them by owner.",
        }
    ],
    max_tokens=8000,
    extra_headers={
        "X-Terminal-SLA-Deadline": "10m",
        "X-Terminal-SLA-Priority": "cost",
    },
    timeout=1800.0,
)

print(response.choices[0].message.content)