Where the Money Goes When You Run an LLM

The first time I sat down with an LLM bill properly, I assumed I was paying for answers. That seemed fair. You ask a question, you get an answer, someone sends you an invoice.

I was mostly paying for the model talking to itself.

Somewhere in the last couple of years, language models picked up the habit of thinking before they speak. It’s a lovely habit. It makes them far better at maths, code and anything with more than one step. It’s also billed by the word, and the words add up faster than you’d guess. So this is a tour of the bill: what you’re actually paying for, which line grew into the biggest one while nobody was looking, where the habit came from, and how to stop it eating your budget.

Your bill, line by line

Every LLM bill is a pile of tokens. A token is a chunk of text, roughly three quarters of a word, and providers charge by the million. Your invoice really only has three kinds.

Input tokens are everything you send: the question, the instructions, the document you pasted in. They’re the cheap ones, because the model reads your whole prompt in one big gulp, and hardware is very good at gulping. Send the same long preamble every time and many providers will cache it and knock about 90% off the repeats. Anthropic, for one, charges a tenth of the normal input price when a cached prompt is reused (Anthropic).

Output tokens are what the model writes back, and they cost several times more: typically five or six times as much at the big providers¹¹Checked against the published price lists at OpenAI, Anthropic and Google. The multiple moves around by model and by month, so treat five or six as the shape rather than the number.. Reading happens all at once. Writing happens one token at a time, and every token is a fresh trip through the entire model. It’s the difference between glancing at a page and copying it out by hand.

Thinking tokens are the new line, and the sneaky one. When a model reasons before it answers, it’s writing, so all of that reasoning is billed as output, the expensive kind. Google doesn’t even pretend otherwise: the line on its price list reads “Output price (including thinking tokens)”. And plenty of providers don’t show you the reasoning you’re paying for. OpenAI’s API doesn’t return the reasoning tokens at all, while Anthropic shows you, at most, a summary written by a different model, and bills you for the full original²²Both document this plainly: OpenAI on reasoning tokens, Anthropic on extended thinking.. You pay for every word either way.

Here’s one question answered three ways. Each version reads the same 1,000-token prompt and writes the same 300-token answer; only the thinking changes. The prices are round numbers I picked to keep the sums easy ($2 per million input tokens, $10 per million output), though they’re in the same ballpark as real mid-range models today.

Look at the bottom bar. The answer you actually read is under 2% of what you paid for. Turning the dial from none to a lot makes the same question 33 times more expensive, and at a million questions a day that’s the gap between $5,000 and $165,000. One of those is a line item. The other is a board meeting.

Thinking out loud, on the meter

“Thinking” is less mysterious than it sounds. A reasoning model doesn’t have a quiet inner voice. It thinks by writing: it scribbles some working-out, tries an approach, spots a mistake, backs up, tries again, and only then writes the answer you see. The working-out is made of tokens like everything else, so the meter runs the whole time.

That’s also why it works. Writing the steps down gives the model somewhere to keep its intermediate results, the way you’d do long division on paper rather than in your head. More room to work means more chances to catch its own mistakes.

Which brings us to the effort setting on most modern models: low, medium, high, sometimes more. It sounds like a request to try harder. What it really sets is how long the model may talk to itself before it has to answer. Low effort is a quick note in the margin. High effort is three pages of working and a double-check.

Think of a taxi with blacked-out windows. The long way round might well be the better route, but the only thing you ever see is the meter.

And the meter runs on the clock as well as the wallet. Writing is the slow part of any model’s job, so every thinking token is time too. OpenAI’s own rule of thumb is that cutting half your output tokens can cut roughly half your wait (OpenAI). Ten thousand words of private working-out is a long, awkward silence before the first word you see.

Why models love the sound of their own voice

Nobody sat a model down and told it to write essays to itself. It picked the habit up in training.

The way most models learn to reason is wonderfully simple. Give them problems with answers you can check, like maths and code, and reward them when they get one right. Nobody marks the working-out. The model just notices, over thousands of rounds, that answers with more working behind them are right more often, so the working grows. When DeepSeek trained its R1-Zero model this way, the average answer on its training problems got steadily longer as training went on, and nobody had asked it to (DeepSeek-AI).

That part is honest: more thinking bought more right answers. But some of the length is an accident of bookkeeping. In GRPO, one of the most popular training recipes, an answer’s reward or penalty gets spread across its tokens. A short wrong answer takes its telling-off in one concentrated dose. A long wrong answer spreads the same telling-off thinly over thousands of tokens, so each one barely feels it. The model learns a lesson every student learns eventually: when you don’t know the answer, write more. One team spotted the quirk and took the averaging out. In one of their runs, accuracy ended up in the same place and answers came out at less than half the length (Liu et al.).

So not every token on your bill is careful thought. Some of it is a nervous habit that training accidentally rewarded, and you pay for it by the word.

The first few thoughts are the bargain

If a little thinking helps, surely a lot helps a lot? Sort of. Thinking has the same shape as nearly everything worth doing: the first bit is hugely useful, and each extra bit is worth less than the one before.

Proofreading works like this. The first read catches the typo in the headline. The second catches a missing word. By the fifth, you’re agonising over a comma you’re not sure you even disagree with.

Take a made-up but honestly shaped example: 1,000 hard questions, with the thinking budget quadrupling at each notch of the dial.

The chart on the right is the one to stare at. Going from off to low buys 150 more right answers for ten dollars. Going from high to max buys 20 more for $480. That final notch costs over 300 times as much per extra right answer as the first.

Real models make the same shape. OpenAI publishes its open-weight gpt-oss-120b scores at all three of its effort settings, and on the 2025 AIME maths paper they run 50.4% at low, 80.0% at medium and 92.5% at high, with the reasoning getting longer at every step (OpenAI). A thirty-point jump, then a twelve-point one, for a much bigger pile of tokens.

None of this makes max effort a scam. Some questions genuinely need the fifth proofread, and for those, the last 2% is the whole point. It just makes max effort a specialist, and paying specialist rates for every question is how bills get weird.

When more thinking makes it worse

That curve flattens out. On some questions it does something ruder: it turns round and heads back down.

Start with the easy stuff. Ask an early reasoning model what 2 + 3 is and it may solve it, check it, solve it another way and check that too. The paper that put this on the map is titled, with feeling, “Do NOT Think That Much for 2+3=?”, and it found reasoning models pouring effort into simple problems for next to no gain (Chen et al.). Most answers contained two to four separate attempts at the same problem, and, gloriously, the models re-solved easy problems more times than hard ones. That’s waste, but at least the answer’s right.

The worse case is when extra thinking talks a model out of the right answer. One study gave leading reasoning models questions a child could answer, like how many fruits you have when you’re holding an apple and an orange, with an irrelevant probability riddle stirred in. For several of the models, the longer they were allowed to reason, the more tangled up they got and the worse they scored (Gema et al.). Other traps went the same way: longer reasoning made models lean harder on misleading patterns in data, and lose the thread in logic puzzles.

It’s the pub quiz team that had the right answer in ten seconds, then spent the next two minutes arguing themselves out of it.

So the effort dial isn’t a volume knob where louder is always better. Past a point you’re not buying care, you’re buying second-guessing, and second-guessing is billed at the same rate.

A bigger brain, or more time to think?

There are two ways to buy a better answer. You can pay for a bigger model, or you can let a smaller one think for longer. It’s the old hiring question: a senior engineer with thirty seconds, or a sharp junior with the whole afternoon?

Price makes it more interesting than it sounds. Small models are several times cheaper per token, so for the cost of a quick think from the big model, the small one gets to think for ages. On plenty of everyday work, the junior with the afternoon catches up with the senior on the clock, and sometimes overtakes them. That isn’t just a hunch. One well-known study found that on questions a small model could already get right some of the time, spending the compute at answer time, letting the small model revise and check its work, beat a model fourteen times its size, compute for compute (Snell et al.).

What the afternoon can’t buy is knowledge. Time helps you use what you know; it doesn’t hand you what you don’t. Some problems need the person who’s seen that exact bug before, and no amount of staring at the screen turns the junior into them. Every small model has a ceiling, and thinking harder only gets you closer to it. The same study found exactly that: on the hardest questions, extra thinking barely helped, and the bigger model was the better buy.

So, a rule of thumb: before you pay for a bigger model, try letting your current one think for longer. Upgrade when the answers stop improving, not the moment they get slow.

Or ask it three times and take a vote

There’s a third way to spend money on an answer, and it’s the oldest trick there is: ask more than once.

Put the same question to the model five times, take whichever answer turns up most often, and you get a better hit rate than any single try. It’s called self-consistency, and it works because wrong answers tend to be wrong in different directions while right ones agree with each other (Wang et al.). The catch is that it only works when answers can be compared, so it suits sums and multiple choice, not essays.

The billing is brutally simple. Five answers cost five times as much, and with a reasoning model that’s five separate rounds of thinking, four of which you paid for and threw away.

So which is the better buy for a fixed budget: one long think, or several quick ones? The study from the last section tested exactly that and found that spending the budget on one answer that revises itself usually beat spreading it across parallel attempts, especially on the easier questions. On the hard ones, a mix of the two won (Snell et al.).

Either way, what you’re buying with the extra calls isn’t brilliance. It’s reliability. Whether that’s worth five times the price is the same question the rest of this post keeps circling: what does a wrong answer cost you?

How a model learns to take a hint

The effort setting itself is humbler than you’d guess. Under the hood it’s often just a note tucked into the model’s instructions. OpenAI’s open-weight gpt-oss models, for instance, take a line like “Reasoning: high” in the system prompt (OpenAI). There’s no special machinery. The model only knows what the note means because it practised with it, over and over, during training.

The practice goes roughly like this:

• A thinking tax. The model earns points for right answers and pays a little for every word of thinking. When the note says low, the tax is steep, so rambling costs more than it earns. When it says high, the tax is tiny, so double-checking pays for itself. After millions of practice problems, it knows what each note is worth.
• Worked examples. The same kind of question shown with a two-line answer under one label and a two-page one under another, like a teacher showing the quick method next to the thorough one. Qwen3 learned its on and off switch partly this way, from a mix of examples tagged “/think” and “/no_think” (Qwen Team).
• Pencils down. Stop the model mid-thought, close its notebook for it, and ask for an answer anyway. Practise that enough and a hard budget stops being a disaster, because the model learns to make the most of whatever it had worked out so far. One method trains on exactly this, cutting the thinking short during practice so the model copes when it happens for real (Xu et al.).

Pencils down has a cheeky mirror image: when the model tries to put its pencil down, don’t let it. The researchers behind s1 simply added the word “Wait” whenever their model tried to stop thinking. It would pause, look back over its work, and often fix a mistake. That alone lifted its score on AIME 2024, a notoriously hard maths competition, from 50% to 57% (Muennighoff et al.). The cheapest management technique in history, now available for machines.

Words like “low” and “high” are fuzzy, so some researchers skip them and write an actual number into the prompt: think for 1,000 tokens, say. The L1 method trains a model to hit that number by rewarding right answers and docking a sliver of reward for every token it misses the target by, too long or too short (Aggarwal and Welleck). The sliver is small but relentless. Miss by more than about 3,300 tokens and a right answer scores lower than a wrong one that arrived on time, a rule every exam invigilator has secretly lived by. It works, too. The trained model lands close to whatever length it’s asked for, and beats simply chopping the thinking off at the limit at every length.

There’s a catch, and it’s a good one. Lean on the tax too hard and the model forgets how to think long at all. It becomes the student who only ever did speed quizzes and freezes in a three-hour exam: quick, cheap, and stuck the moment a problem needs stamina. So labs mix things up. Kimi’s team alternates between strict practice on a budget and relaxed practice without one, and reports cutting its model’s output by 25 to 30% with barely any loss in quality (Kimi Team). An earlier Kimi model tried something that sounds too simple to work: train a brisk model and a thorough one separately, then average their weights into a single model (Kimi Team). It apparently does.

It’s also why typing “please think really, really hard” into a model that never practised with an effort note doesn’t get you the same thing. The note is a light switch someone wired up during training. Writing “max” on a model that was never wired for it is painting a switch on the wall.

And even a properly wired switch has a price. Asking one model to be both a sprinter and a marathon runner is a compromise, and at least one lab has since gone back to separate models. When Qwen refreshed its flagship in 2025, it shipped two: one that only ever thinks, and one that never does³³The two model cards: the thinking one and the one that doesn’t.. On Qwen’s own table, the never-thinks model scored 70.3 on the AIME 2025 maths test, against 24.7 for the old switchable model with its thinking turned off. It’s a newer model, so that isn’t a clean experiment. But nobody rips out a switch that was costing them nothing.

Asking isn’t the same as getting

An effort setting is a request, not a contract. Anthropic’s documentation says plainly that its thinking budget is a target rather than a strict cap, and that the model may stop well short of it (Anthropic). Researchers who measured how much models actually think when handed a budget found the same looseness from the other side. Bigger budgets do mean longer thinking on average, but individual answers scatter a long way from what was asked (Gema et al.).

That scatter is the part averages hide. Depending on the problem, OpenAI says its reasoning models may write anything from a few hundred to tens of thousands of reasoning tokens (OpenAI). So your cost per call isn’t a number, it’s a spread, and your slowest users live in the long tail of it. A dashboard showing the average will look perfectly calm while a handful of requests quietly take ages and cost a small fortune.

If you need a hard stop, most APIs have one: a cap on total output tokens. Just know what it does. If the cap lands mid-thought, you can get an empty or half-finished answer and still pay for every token of thinking that came before it⁴⁴See OpenAI and Google on what a budget actually promises. Google’s is a target, not a cap.. Google’s own advice is to turn the thinking level down rather than set a small cap. Keep the cap as a safety net, high above normal use, and do the everyday steering with the effort dial.

Letting the model choose

The obvious next step is to stop picking the effort yourself and let the model judge each question. This already exists. Gemini models think dynamically by default, adjusting how much they reason to how complex the request is (Google). Claude has an adaptive mode that decides when and how deeply to think, and skips thinking altogether on simple requests (Anthropic). Some of the choosing happens outside the model entirely: in ChatGPT, GPT-5 shipped as a system of thinking and non-thinking models with a router deciding which one takes your question (OpenAI).

Research models show how the judgement can be taught. AdaptThink trains a reasoning model to choose, question by question, between thinking first and answering straight away. On three maths benchmarks it cut the smaller model’s average answer length by 53% and nudged its accuracy up at the same time (Zhang et al.). Thinkless does something similar with two special tokens, one that starts a short answer and one that starts a long think, and cut long-form thinking by 50 to 90% on several maths benchmarks (Fang et al.).

Letting the model choose is a lot like letting a taxi driver choose the route: usually sensible, occasionally scenic, and you’re still the one paying. Keep a ceiling on it, and keep an eye on the meter.

The cost that never shows up on the invoice

There’s one more cost, and you’ll never see it as a line item. While a model writes, it keeps a running memory of the conversation so far, called the KV cache, in the GPU’s memory. That memory grows with every token, and it stays put until the request is done. Long thinking means a big slice of memory, held for a long time.

GPU memory is the scarcest thing in the building. In the paper behind vLLM, one of the most widely used serving systems, a 13-billion-parameter model on a 40GB GPU took up about 65% of memory by itself, leaving roughly 30% for every request’s KV cache put together (Kwon et al.). A request that thinks for ages hogs room that other requests could have used, so the GPU serves fewer people at once, and each of them costs more to serve.

You pay for that indirectly. It’s part of why writing is priced above reading in the first place, and why some providers charge more per token once a request gets long: OpenAI’s price list has a separate, pricier column for long context (OpenAI). Others, like Anthropic on its recent models, charge one flat rate across the whole million-token window (Anthropic). Either way, a model that thinks for 50,000 tokens isn’t just running up the word count. It’s taking up three seats on a very expensive bus.

The only number worth watching

Most dashboards show cost per call. It’s the wrong number, because a cheap call that gets it wrong isn’t cheap. You either pay for another call or you pay for the mistake, and mistakes are rarely billed in fractions of a cent.

The number worth watching is cost per good answer:

def cost_per_good_answer(cost_per_call, hit_rate, cost_of_a_miss=0.0):
    # On average you make 1 / hit_rate calls to get one good answer,
    # and every miss along the way also costs whatever the mistake does.
    return (cost_per_call + (1 - hit_rate) * cost_of_a_miss) / hit_rate

a_little = dict(cost_per_call=0.015, hit_rate=0.70)
a_lot = dict(cost_per_call=0.165, hit_rate=0.84)

cost_per_good_answer(**a_little)                    # 0.021  misses are free
cost_per_good_answer(**a_lot)                       # 0.196
cost_per_good_answer(**a_little, cost_of_a_miss=1)  # 0.450  misses cost $1
cost_per_good_answer(**a_lot, cost_of_a_miss=1)     # 0.387

Those are the two settings from earlier: a little thinking gets 70% right, a lot gets 84%. If mistakes cost you nothing (you can check an answer instantly and just ask again), a little thinking works out about nine times cheaper per good answer. Keep it quick and try again.

Put a price on being wrong, though, even a dollar for someone to notice and fix it, and the picture flips: the expensive setting is now the cheaper one. In this toy example, the crossover comes when a mistake costs about 75 cents.

Drag the accuracy sliders rather than the price one and you’ll see the uncomfortable part: a couple of points of hit rate move the crossover further than a tenfold change in the price of a call would.⁵⁵Which is an argument for measuring your hit rate before you shop for a cheaper model. Most teams know their per-call price to four decimal places and their hit rate not at all.

Whether thinking is a bargain depends almost entirely on what a mistake costs you.

A clumsy sentence in a draft email costs nothing. A wrong total on an invoice, a bad database migration or a confidently wrong reply to a customer costs a great deal. Price the thinking against that, not against the cheapest possible call.

Spending it wisely

A few habits that keep the bill honest:

• Pick effort per task, not per app. Summarising a meeting and debugging a race condition don’t deserve the same budget. One global setting is nearly always either wasteful or stingy.
• In agents, pick it per step. An agent that plans, calls tools and tidies up results is doing very different kinds of work. Planning deserves real thought. Turning a tool’s output into JSON deserves roughly none. A 40-step run with 20,000 thinking tokens on every step costs $8 in thinking alone at our made-up prices. Give the four planning steps the big budget and everything else 1,000 tokens, and the same run costs about $1.16. Maxing out every step is hiring a structural engineer to hang a picture frame: beautifully reasoned, and it’s still just a nail.
• Keep thinking and talking separate. How long the model thinks and how long its answer is are two different dials. If you want a shorter answer, ask for a shorter answer rather than starving the model of thinking time. GPT-5 even has a separate verbosity setting for the length of the answer, apart from its reasoning effort (OpenAI).
• Know where old thinking goes. In long chats and agent runs, check what happens to earlier reasoning. Some models drop it after each turn. Others carry it into the next turn’s context, where (on Claude, for one) it’s billed as input like the rest of the conversation, turn after turn⁶⁶Anthropic and OpenAI each describe what happens to a previous turn’s reasoning. They do not do the same thing, which is the point.. Carried-over thinking can make the model sharper, but it’s a cost that compounds.
• Set a budget, and let it land. A thinking budget is a timeout by another name. Choose it on purpose, and favour models that wrap up gracefully when time’s up over ones that get cut off mid-sentence.
• Keep an eye on the thinking share. Most APIs report how many of your output tokens were thinking, right there in each response’s usage details⁷⁷The usage object on every response carries it: OpenAI, Anthropic.. If it’s creeping past 90% on work that feels easy, some part of your system is paying a specialist to do data entry.

Thinking is the best thing to happen to these models in years, and I wouldn’t give it back. It’s just billed like a lawyer who thinks out loud. Point it at the problems that deserve it, keep it brief everywhere else, and glance at the meter now and then.

The cheapest token is the one you never needed. The priciest is the one that thought very hard and still got it wrong.

Sources

Every claim above about a real model, price or paper comes from one of these. Prices are taken from the providers’ own price lists, which change often. The six figures marked “From the paper” are reproduced unmodified under their papers’ CC BY 4.0 licence and load straight from arXiv. The seven marked “Illustrative” are mine, drawn from made-up numbers.

Prices and documentation

• OpenAI, API pricing
• Anthropic, Pricing
• Google, Gemini Developer API pricing
• OpenAI, Reasoning models
• OpenAI, Latency optimization
• OpenAI, Introducing GPT-5 for developers
• OpenAI, gpt-oss-120b model card and the gpt-oss model card in full (PDF)
• Anthropic, Thinking
• Anthropic, Extended thinking
• Anthropic, Prompt caching
• Google, Gemini thinking
• Qwen, Qwen3-235B-A22B-Instruct-2507 and Qwen3-235B-A22B-Thinking-2507 model cards

Research

• DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, 2025
• Liu et al., Understanding R1-Zero-Like Training: A Critical Perspective, 2025
• Chen et al., Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs, 2024
• Gema et al., Inverse Scaling in Test-Time Compute, 2025
• Wang et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models, 2022
• Snell et al., Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters, 2024
• Qwen Team, Qwen3 Technical Report, 2025
• Xu et al., Scalable Chain of Thoughts via Elastic Reasoning, 2025
• Muennighoff et al., s1: Simple test-time scaling, 2025
• Aggarwal and Welleck, L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning, 2025
• Kimi Team, Kimi k1.5: Scaling Reinforcement Learning with LLMs, 2025
• Kimi Team, Kimi K2.5: Visual Agentic Intelligence, 2026
• Zhang et al., AdaptThink: Reasoning Models Can Learn When to Think, 2025
• Fang et al., Thinkless: LLM Learns When to Think, 2025
• Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, 2023