AI & Machine Learning for Developers
Working with LLM APIs
Most developers don't train a language model — they call one that's already running behind an API, hosted by one of several providers. That interaction has its own vocabulary and constraints that don't come up in a typical REST API: prompts instead of fixed parameters, tokens instead of characters as the unit of cost and length, and streaming responses instead of one complete reply. This guide covers those mechanics generically; multiple vendors offer hosted LLM APIs with broadly similar shapes, and the specific request format differs by provider.
Why it matters
- Tokens, not characters or words, are the real unit
- Pricing, context limits, and rate limits are all measured in tokens (roughly, pieces of words), so a request that looks short in characters can still be a large number of tokens, and vice versa.
- The prompt is effectively the entire program
- Unlike a typical API call with a handful of fixed parameters, most of the behavior of an LLM API call is controlled by the text of the prompt itself, which makes prompt content a first-class part of the system to test and version, not just a string.
- Streaming changes how you build the calling application
- Many LLM APIs can return a response piece by piece as it's generated rather than waiting for the whole thing, which matters for anything user-facing, where showing a growing response feels far more responsive than a long blank wait.
- Rate limits and cost scale with usage in a way many APIs don't
- Because both cost and throughput are tied to token volume, a feature that works fine in testing can hit limits or become expensive in ways that only show up once it's used at real scale.
Prompts, tokens, and context length
A request to an LLM API is built around a prompt — text (and sometimes other content) describing what you want the model to do, often alongside a running conversation history. Internally, that text is broken into tokens, which are typically smaller than whole words for less common terms and can be a whole word or more for common ones; the exact tokenization scheme is specific to each model. Two numbers set the boundaries of what's possible in a single call: the context length (the maximum number of tokens the model can consider at once, covering both the prompt and the reply) and, usually, a separate limit on how many tokens the reply itself can contain. Going over the context length doesn't get a graceful partial answer — it's a hard limit, which is why long conversations or large retrieved documents (see this section's RAG guide) often need to be trimmed or summarized to fit.
Streaming, rate limits, and calling the API from code
A typical hosted LLM API can be called in two modes: wait for the full response, or stream it back incrementally as it's generated, token by token or in small chunks. Streaming is usually the better choice for anything a person is watching in real time, since it shows visible progress instead of a long pause. Almost every provider also enforces rate limits — a cap on requests or tokens over a period of time — and returns a specific error when a caller goes over that cap, which application code needs to handle explicitly (typically by waiting and retrying) rather than treating as a generic failure. Because a single response can take a meaningful amount of time to fully generate, and can fail partway through, production code calling these APIs generally needs retry logic and timeouts in a way a fast, typical REST call often doesn't.
import requests
def ask_model(prompt, api_key, api_url):
response = requests.post(
api_url,
headers={"Authorization": "Bearer " + api_key},
json={
"prompt": prompt,
"max_output_tokens": 300,
"stream": False,
},
timeout=30,
)
response.raise_for_status()
return response.json()["output_text"]
# Real providers differ in exact field names and request shape;
# check the specific provider's documentation before calling it.Mistakes people make here
- Storing an API key directly in source code
- A key committed to version control or hardcoded in a client-side application is effectively public; keys belong in environment variables or a secrets manager, and calls that need to stay private belong on a server, not in a browser.
- Not handling rate-limit errors explicitly
- A burst of user traffic will eventually hit a rate limit; without retry-with-backoff logic, that shows up to users as a hard failure instead of a brief delay.
- Assuming token count equals word count
- Token counts and word counts diverge, sometimes significantly, especially with punctuation, non-English text, or unusual formatting, so estimating cost or context usage from a word count alone is unreliable.
- Treating the prompt as a one-off string instead of something to version and test
- Since the prompt drives most of the model's behavior, changing its wording changes the application's behavior; teams that don't track prompt changes the way they track code changes lose the ability to explain why behavior shifted.
- Not setting a timeout or handling a partial or failed generation
- Generation can take a variable amount of time and can fail partway through; code that assumes every call either fully succeeds quickly or throws immediately will handle real-world latency and errors poorly.
Strengths and trade-offs
Where it is strong
- No need to train, host, or maintain a model yourself — a working system can be built entirely on calling one over an API.
- Streaming responses make it practical to build interfaces that feel responsive even though full generation takes real time.
- The general shape (prompt in, generated text out, usage measured in tokens) is similar enough across providers that the core concepts transfer even when the exact request format doesn't.
The trade-offs
- Cost and rate limits scale directly with usage in a way many traditional APIs don't, so testing at small scale can hide problems that appear at production volume.
- Sending data to a hosted API means that data leaves your own infrastructure, which is a real consideration for private or regulated data regardless of the provider's own security practices.
- Behavior can shift when a provider updates the underlying model, even without any change to your own code.
- Latency is generally higher and more variable than a typical database or REST call, which has to be designed for rather than assumed away.
Who needs this
Any developer building a feature that calls a hosted language model needs this — the mechanics here (tokens, context limits, streaming, rate limits) affect the design of the calling code regardless of which specific provider is used. If you're only using an AI product as an end user rather than building against its API, this is not necessary.
Questions about working with llm apis
- Do all LLM API providers work the same way?
- The broad shape — send a prompt, get generated text back, pay and get limited based on tokens — is common across providers, but the exact request format, parameter names, and pricing differ, so the specific provider's own documentation is still necessary.
- What exactly is a token?
- Roughly, a chunk of text a bit smaller than or equal to a word, produced by a model-specific tokenization scheme; the precise breakdown varies by model and isn't something to hand-calculate, but it's the unit almost every limit and price is measured in.
- Is streaming always the right choice?
- Not always. It suits interactive, user-facing use where perceived responsiveness matters. A backend job processing many requests in bulk often has no one watching in real time and can simply wait for complete responses.
- Is it safe to call these APIs directly from a browser?
- Generally no, if it requires an API key — a key embedded in client-side code is exposed to anyone who inspects the page. The typical pattern is to call the API from a server you control, which keeps the key private.