Cost guide

Hidden Costs of LLM APIs Beyond Per-Token Pricing

· · Estimated read: 7 min

Headline per-token prices only tell part of the story. Production LLM applications incur costs from hosted tools, rate limits, latency, fine-tuning, evaluation, integration, and vendor management — often exceeding the token cost itself. This guide maps the full cost landscape so you can estimate total cost of ownership.

1. Fine-Tuning Costs

Fine-tuning an LLM on your data adds three cost layers on top of base token pricing:

  • Training tokens. Providers charge per token for training. For example, fine-tuning a model on a 100k-token dataset for 3 epochs means processing 300k training tokens — at rates that can be 10-50x higher than inference pricing.
  • Data preparation. Cleaning, labeling, and formatting training data requires engineering time. Poor data quality leads to repeated fine-tuning runs, multiplying costs.
  • Hosting fine-tuned models. Some providers charge hourly or monthly for hosting a private fine-tuned model, separate from inference costs.

What to do: Estimate fine-tuning cost before starting. Factor in data preparation time and expected iteration count. For small datasets, prompt engineering with a high-quality base model may be cheaper than fine-tuning.

2. Throughput and Rate Limits

Token prices are the unit cost, but your effective cost depends on how many tokens you can process per minute (TPM) and requests per minute (RPM).

  • Free tier limits. Many providers advertise free or low-cost tiers with strict TPM/RPM caps. Exceeding these requires upgrading to paid tiers — which may have higher per-token rates or monthly commitments.
  • Rate limiting costs. When you hit rate limits, requests fail or queue. Retries consume tokens on partial responses. Backoff delays increase latency, affecting user experience and requiring more concurrent connections.
  • Batch vs real-time. Batch APIs are often 50% cheaper than real-time, but introduce latency hours to days. Our cache and batch pricing guide covers this in detail.

What to do: Check provider rate limit docs early. Use the calculator with the batch pricing preset to compare batch vs real-time costs for your workload volume.

3. Latency Cost Tradeoffs

The cheapest model isn't always the cheapest overall — slow responses cost user engagement, compute time, and connection resources.

  • Smaller vs larger models. A $0.15/M token model may be 5x slower than a $0.50/M token model for the same task. The cost of user wait time (abandonment, reduced throughput) can outweigh token savings.
  • Streaming overhead. Streaming responses improve perceived latency but use more connection resources and may incur higher egress costs depending on the provider.
  • Cold starts. Models that auto-scale from zero have cold-start latency (1-10 seconds for first request), requiring warm instances for production — at additional cost.

What to do: Use the compare page to compare prices and model capabilities, then measure response latency on your own representative requests. For latency-sensitive applications, test both cheap and mid-range models — the faster model may deliver better overall value.

4. Evaluation and Testing Costs

Running evaluations to measure quality consumes tokens — and comprehensive test suites add up quickly.

  • Eval datasets. Running 1,000 test cases through a model at 2k tokens each costs roughly $4-20 per eval run (depending on model tier). Multiply by iterations during prompt engineering.
  • Regression testing. After any model update or prompt change, re-running the full test suite is necessary to catch regressions. This is an ongoing cost, not a one-time expense.
  • A/B and shadow testing. When A/B traffic is split between models, each request normally goes to one model; total inference spend depends on the traffic split and each model's price. Shadow testing that sends the same request to a second model adds a separate inference charge for that duplicate request.

What to do: Budget eval costs as a recurring operational expense. Use the benchmark explorer to understand model capabilities before investing in custom evaluation.

5. Integration and Operational Overhead

Building a reliable LLM pipeline involves engineering costs that don't appear on any provider invoice but affect your total cost.

  • Multi-provider abstraction. Writing and maintaining routing logic, fallback chains, and model-specific retry handling requires ongoing engineering investment. Our model routing cascade guide covers strategy but implementation cost varies.
  • Observability. Logging tokens, tracking cost per user or feature, monitoring latency, and detecting quality regressions — all require instrumentation and dashboard infrastructure.
  • Prompt management. Versioning prompts, managing prompt templates across environments (dev/staging/prod), and coordinating prompt changes across teams adds process overhead.

Direct provider API or routing gateway?

If your workload has outgrown a single-provider setup, compare the operational trade-off before adding a routing layer.

Decision factor Direct provider API Routing gateway such as OpenRouter
Stable single-provider workload Shorter path when one provider and its native account controls are enough. Useful when one endpoint needs to span multiple providers or models.
Rate limits and outages Your application owns retry and fallback logic. OpenRouter can fall back across providers and, when configured, across an ordered model list.
Total cost You pay the provider directly without a separate OpenRouter platform fee. OpenRouter charges provider-listed inference rates without a per-request markup. Its fee is charged when buying credits: 5.5% on Standard or 8% on Business. BYOK has separate allowance and fee terms. See OpenRouter pricing.
Cache-sensitive agent sessions Keeping requests on the same provider endpoint avoids routing drift between turns. Use sticky routing with a stable session ID when warm prompt-cache continuity matters.

For a single provider with predictable limits, direct access is usually the simpler operational path. A routing gateway becomes more valuable when failover, multi-provider access, or endpoint selection is worth the additional platform layer and fee. For cache-heavy agents, treat route stability as part of the cost model.

Current routing details: OpenRouter pricing, provider routing, model fallbacks, and sticky prompt caching.

6. Hosted Tool Calls and Runtime Charges

Agent workflows can incur charges that are metered separately from model tokens. A request that invokes a provider-hosted tool can therefore have more than one billable component even when the model's input and output token counts are known.

  • Tool calls can be a separate meter. OpenAI's current API documentation notes that tool-specific features such as web search and file search can carry a per-tool-call fee in addition to model token charges.
  • Hosted containers have a different meter. OpenAI Hosted Shell and Code Interpreter can charge for container session duration based on memory size, separately from model tokens. Record the container tier and billed session minutes; prompt counts and tool-call counts do not describe this cost.
  • Usage responses may expose tool activity separately. Anthropic's web-search documentation reports search usage under server_tool_use.web_search_requests and bills that usage separately from the model tokens consumed by the request. Search-result content can also add input token charges.
  • Google Gemini uses a different search unit. In the Gemini Developer API, Gemini 3 and newer Google Search grounding counts individual search queries, not user prompts: one prompt can produce multiple searches. Gemini 2.5 instead bills by grounded prompt. The newer models share a monthly free search allowance, so the number of searches in one response does not establish the billable total. See Google Search grounding and Gemini API pricing.
  • DeepSeek Search and Fetch are billed per successful request. Each API costs $0.75 per 1,000 successful requests, separately from LLM tokens. A Search request is billed only when it returns results; a Fetch request only when it returns page content (both require HTTP 200). For the DeepSeek API used through DeepSeek Harness, Claude Code, or Codex, DeepSeek announced that its temporary exemption for server-side Search/Fetch calls would end at 00:00 Beijing time on November 1, 2026. That date has passed: check current usage charges rather than assuming those calls are still free. Standalone Search/Fetch API requests were not covered by the exemption. See DeepSeek pricing and billing rules.
  • Agent loops multiply both meters. If one user request triggers several searches or other hosted-tool invocations, counting only prompt and completion tokens understates the request's total API cost.

Compare billing units before comparing costs. A user prompt, an API request, a hosted search invocation, and an underlying search query are not interchangeable counts. Check the tool version, search usage, and any search-result token charges on each provider's billing surface.

What to do: Track hosted-tool invocations and, for hosted containers, session minutes and memory tiers alongside input, cached-input, and output tokens. Budget each meter separately and set search-use limits when available. The Calculator's actual-usage mode can include reported OpenAI Web Search calls and hosted-container usage. The Gemini importer can show observed search-query counts, but cannot know your project's remaining shared free allowance, so it does not automatically bill Gemini grounding. Reconcile any other unpriced search and non-token charges against provider usage or invoices. DeepSeek Search/Fetch fees are not included; budget them separately using successful request counts.

Current provider references: OpenAI API tool pricing and Claude API tool pricing.

7. Vendor Risk and Migration Costs

Switching LLM providers involves more than changing an API key. These migration costs are often underestimated.

  • Prompt re-engineering. Prompts optimized for one model rarely work optimally on another. New provider means new prompt testing and tuning — sometimes from scratch.
  • Embedding regeneration. If you use provider-specific embedding models, changing providers requires re-embedding your entire document corpus — costing tokens and engineering time.
  • Contract lock-in. Volume discounts and committed-use contracts lock you into a provider. Breaking these or over-committing wastes budget.

What to do: Design for portability from day one. Keep prompts in version control, abstract model selection behind a simple interface, and prefer providers with compatible API surfaces.

8. Compliance and Data Residency

Enterprise requirements around data handling carry cost implications that per-token pricing doesn't capture.

  • Regional pricing. US-based API pricing is often the cheapest. EU, Asia, or sovereign cloud regions may add 20-50% premium on token costs.
  • Data processing agreements. HIPAA BAA, GDPR DPA, or SOC2 reports may require specific provider tiers or enterprise plans with higher minimum commitments.
  • Data egress. Hybrid architectures that route some requests locally and some to the cloud incur bandwidth costs for both directions.

What to do: Check compliance docs before building on a provider. Factor regional pricing differences into your cost model. See our data sources page for provenance information on provider pricing data.

Before comparing an estimate with your bill

Read the API response's usage counters before multiplying token counts by catalog rates. Providers group cached and generated tokens differently, so adding every reported counter together can overstate the cost.

  • OpenAI Responses: input_tokens already includes cached input; input_tokens_details.cached_tokens is a subset, not extra input. output_tokens includes non-visible generated tokens, including reasoning where applicable.
  • Anthropic Messages: input_tokens, cache_creation_input_tokens, and cache_read_input_tokens describe separate input categories. Count each once at its applicable rate rather than treating input_tokens as the full prompt.
  • Gemini: promptTokenCount includes cachedContentTokenCount, so do not add cached tokens again. candidatesTokenCount and thoughtsTokenCount are separate generated-token counts; tool activity may need its own accounting.

Use the Calculator's actual-usage mode to enter or import supported OpenAI, Anthropic, and Gemini usage records. Match the exact model, route, usage date, and processing tier; then check hosted-tool fees, discounts, credits, taxes, and other invoice-only items separately. The calculator estimates modeled charges; it does not reconcile your provider invoice.

Usage references: OpenAI token counting, Anthropic cache counters, and Gemini usage metadata.

Summary

Per-token pricing is the visible cost, but the hidden costs — fine-tuning, hosted-tool fees, rate limits, latency tradeoffs, evaluation, integration, migration, and compliance — often dominate total cost of ownership for production LLM applications.

Use our calculator to estimate per-token costs, then layer in these hidden factors. The right model choice depends on your full cost picture, not just the cheapest headline price.

Pricing data sourced from our catalog. Check data sources for provenance and freshness.