Every article tagged LLM, in one place.
Showing 19 articles
Large language model APIs all follow the same basic shape: you send a list of messages and get a message back. This guide uses the official Anthropic SDK to call Claude, but the ideas carry over to any provider. 1. Install the SDK and set your key # TypeScript...
An LLM only knows what was in its training data. Retrieval-augmented generation (RAG) lets it answer questions about your data, such as docs, tickets or policies, by finding the relevant passages first and putting them in the prompt. The pipeline Ingest: split...
"Agent" is used for many things, but the core idea is simple: an AI agent is an LLM that decides which tools to call, sees the results, and keeps going until the task is done . Your code runs the tools; the model chooses what to do next. The loop Send the user...
Before MCP, every AI application needed custom glue code for every integration: one plugin for GitHub in app A, a different one in app B. The Model Context Protocol (MCP) is an open standard, introduced by Anthropic, that defines one way for AI applications to...
Long answers can take many seconds to generate. Streaming shows text as it is produced, so users see progress immediately instead of staring at a spinner. Streaming on the server The SDK's messages.stream() helper gives you an async iterator of events plus a t...
The system prompt is the brief you give the model before the conversation starts. Most poor LLM results come from a vague brief, not a weak model. 1. Give context, not just commands Explain who the users are and why the task matters. A model that understands t...
Asking a model to "reply in JSON" works most of the time. Production code needs it to work every time. Structured outputs constrain the response to a JSON Schema, so the result always parses. TypeScript with Zod import Anthropic from '@anthropic-ai/sdk'; impor...
Many apps send the same large prefix on every request: a long system prompt, tool definitions, or a document the user is asking about. Prompt caching stores that prefix so later requests read it from cache at a fraction of the normal input price and with lower...
LLM APIs bill by the token , a chunk of text that is often a word or part of a word. Input and output tokens are priced separately, and output tokens cost several times more than input tokens. Count tokens before you send Token counts depend on the model's tok...
LLM APIs are network services under heavy load, so failures are normal. The trick is knowing which errors to retry and which to surface to the user. Retryable vs non-retryable Retry: 429 rate limit, 5xx server errors, overloaded errors, timeouts and dropped co...
Users trust an answer more when they can click through to the source. Citations also make hallucinations easy to spot: an unsupported claim has nothing to point at. Option 1: Ask for chunk IDs Answer the question using only the sources below. After each senten...
Few-shot prompting means showing the model a handful of input/output examples before the real input. It is the fastest way to teach a specific style or labelling scheme without fine-tuning. A classification example Label each commit message as feat, fix, docs...
Without evaluation, every change to chunking, embeddings or prompts is a guess. A small labelled dataset turns RAG tuning into engineering. 1. Build an evaluation set Collect 50-200 real questions from logs, support tickets or users. For each, record the ID(s)...
Every turn of an agent adds to the conversation: tool calls, large tool results, the model's replies. Long tasks can grow past the context window, and even before that, a bloated context makes the model slower, more expensive and less focused. 1. Keep tool res...
Some responses shouldn't wait until everything is ready: large exports, progress updates, or text from an LLM. FastAPI's StreamingResponse sends data as your generator produces it. Streaming a large CSV from fastapi.responses import StreamingResponse @app.get(...
Reasoning models can "think" before answering: they work through a problem internally, then write the final response. Thinking improves results on hard tasks such as multi-step maths, debugging and planning, at the cost of extra tokens and time. Adaptive think...
Agents are exciting, but they cost more, are slower and are harder to test than fixed code. Many features are better built as workflows : LLM calls arranged in a flow that your code controls. Start with the simplest thing One LLM call - classification, summari...
Modern LLMs read images and PDFs as well as text. That makes it easy to build features like "extract the line items from this invoice" or "explain this error screenshot". Image from a URL const response = await client.messages.create({ model: 'claude-opus-5-5'...
LLM features fail in new ways: answers get worse without any error, costs spike because a prompt grew, an agent loops through tools. Normal API monitoring doesn't catch these, so add LLM-specific telemetry. 1. Usage and cost const response = await client.messa...
