What for: ultra-fast inference of open models on LPU hardware — minimal response latency for real-time voice agents and streaming chats.

What it is and who it's for

Groq is cloud inference for open models on its own LPU hardware, delivering record speed and minimal latency. For engineers building real-time scenarios: voice agents, streaming responses, "instant" assistants where responsiveness matters more than intelligence records. What sets it apart: it pushes hundreds of tokens per second — the answer "types itself" almost instantly.

Key features

Ultra-low latency and high throughput on LPU (tens→hundreds of tokens/s).
OpenAI-compatible API — migrate by swapping the base URL and key.
A catalog of open models (Llama, Qwen, Mixtral, Whisper for STT and others).
Token streaming for a "live" UX in chat and voice.
A free tier for development and generous limits to start with.
Function calling / tools on supported models.

Getting started in 5 minutes

Sign up at console.groq.com and get an API key.
Use the OpenAI-compatible endpoint:

curl https://api.groq.com/openai/v1/chat/completions \
  -H "Authorization: Bearer $GROQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"llama-3.3-70b-versatile","messages":[{"role":"user","content":"Hello"}]}'

Set "stream": true to see the answer "type" instantly.

When to take it and when not to

✅ Take it if: you're building a voice agent or a chat where latency kills the UX, and open models are good enough.
✅ Take it if: you need cheap, very fast STT (Whisper) or fast streaming summarization.
❌ Skip it → go with "OpenRouter" if you need top proprietary models (Claude/GPT) or the widest possible choice.
❌ Skip it → go with "Together AI" if you need fine-tuning and serverless training of open models.

The honest price

There's a free tier with limits; paid usage is per token, and prices for open models are usually competitive. Top throughput may be billed separately. Exact amounts are on the provider's side and change.

Gotchas

The catalog is mostly open models; the smartest proprietary frontier isn't here.
The free tier has rate limits; production traffic needs a paid plan and quota monitoring.
Availability of a specific model and its context window change — pin them in config and watch for deprecations.
The speed is impressive, but answer quality is capped by the open model you pick.

🤖 Prompt accelerator

"I'm building a <voice agent / streaming chat> and I'm using Groq for low latency. Pick an open model from their catalog for my task and language <RU/EN>, give me a streaming request example in <Python/Node>, suggest parameters (temperature, max_tokens) and how to measure time-to-first-token. If I need STT — suggest a Whisper model and an example."