The Energy and Water Footprint of AI: What Training and Inference Really Cost

Every few months, a new headline claims a single chatbot query drinks a bottle of water or burns as much power as a light bulb running for an hour. The next headline says the opposite. Both can cite a real study. The honest answer is that AI’s resource footprint is real, growing, and genuinely difficult to pin to a single number, because the number depends heavily on assumptions most articles don’t state. That’s not a reason to dismiss the concern — it’s a reason to understand where the energy and water actually go, and what, if anything, a practitioner running their own infrastructure can do about it.

Where the Energy Actually Goes

It helps to separate three genuinely different costs that get lumped together in casual conversation.

Training is a large, front-loaded cost. A frontier model’s training run occupies thousands of accelerators running continuously for weeks or months. This is a real, large draw on a data center’s power budget, concentrated in a short window, and it happens once (or a handful of times, across checkpoints and re-training) per model.

Inference is the opposite shape: individually tiny, but it happens every single time anyone sends a query, for as long as the model stays in production, which for a popular model is billions of times. A model that trains once and then serves a billion queries a month has its lifetime energy use dominated by inference, not training, even though a single training run looks far more dramatic in isolation.

Embodied cost is the energy and materials that went into manufacturing the hardware itself — fabricating the accelerator chips, building the servers, constructing the data center — before a single watt is spent computing anything. This category is easy to forget because it doesn’t show up on a monthly electricity bill, but chip fabrication in particular is an energy- and water-intensive industrial process in its own right, and it’s a real part of AI’s total footprint regardless of how efficiently the resulting hardware is later used.

Cost category	Shape	Dominant when
Training	Large, one-time, concentrated	Model is niche or rarely queried after training
Inference	Small per-query, continuous, aggregates	Model is popular and served at scale (the common case)
Embodied (manufacturing)	Fixed per unit of hardware, amortized over its service life	Hardware is replaced frequently or underutilized

Training: Genuinely Large, But Not the Whole Story

A frontier-scale training run is, in absolute terms, one of the larger single computing workloads that exists — comparable to running a mid-size data center’s worth of accelerators flat out for an extended period. That much is well established and not seriously disputed. What’s contested is how to fairly attribute that cost per unit of downstream value: a training run that produces a model used by hundreds of millions of people amortizes very differently, per user, than one used internally by a single team. Precise kilowatt-hour figures for any specific model’s training run are rarely published with enough methodological detail to independently verify, so treat any headline number here as an estimate, not an audited fact.

Inference: Small Per-Query, Enormous in Aggregate

A single inference request — one prompt, one response — costs a small, genuinely modest amount of energy compared to a training run. The trouble is that “small” and “aggregate” don’t compose the way intuition suggests. A cost that looks negligible at the scale of one user’s one question stops looking negligible when it’s multiplied by the query volume of a service used by a meaningful fraction of the internet, run continuously, forever.

This is also the category where model choice matters most in practice: a larger model with more active parameters costs meaningfully more energy per token generated than a smaller one, which is one of the more actionable levers discussed below.

Embodied Carbon: The Part Nobody Sees on the Bill

Chip fabrication is a resource-intensive industrial process — it uses significant energy and, notably, ultrapure water for wafer cleaning, at volumes that are substantial for the semiconductor industry as a whole. None of this shows up in a conversation about “how much power did my query use,” because it happened long before the chip was installed anywhere, but it’s a real part of the total picture, and it’s part of why hardware utilization and replacement cadence matter: a GPU sitting mostly idle, or one retired well before the end of its useful life, is embodied cost that never got amortized into useful work.

Water: Cooling, Not Drinking

Data centers use water primarily for cooling, not because computers consume water the way people do, but because evaporative cooling towers are an efficient way to reject the heat that racks of accelerators generate continuously. Water absorbs heat and some of it evaporates off, and that evaporated water is what shows up in a data center’s reported “water usage.”

The reason per-query water estimates vary so wildly between sources comes down to a handful of methodological choices that rarely get stated explicitly in the headline number:

• Which cooling technology the specific data center uses. Evaporative cooling towers use meaningfully more water than closed-loop liquid cooling or air-cooled designs, and the mix varies enormously by facility and by region’s climate.
• Regional water stress and source, which affects how much a given volume of water actually matters locally — water drawn from an already-stressed aquifer is a different problem than water drawn from an area with abundant rainfall, even if the raw volume is identical.
• On-site vs. off-site water use. Some estimates count only water used directly at the data center; others also fold in the water used to generate the electricity it consumes (thermal power plants use water for cooling too), which can dominate the total depending on the local grid mix.
• PUE and WUE as reported metrics. Power Usage Effectiveness and Water Usage Effectiveness are the standard industry metrics, but they’re self-reported by operators with varying disclosure practices, and they describe a facility average, not any individual workload.

Given all of that, a specific “milliliters per query” figure you see quoted should be read as one estimate under one set of assumptions, not a physical constant. The direction of the trend — more AI usage means more aggregate water draw for cooling, somewhere in the pipeline — is well established even where the precise multiplier is not.

Why Per-Query Numbers Are So Contested

Beyond cooling specifically, the same handful of variables explain most of the disagreement in energy-per-query numbers too:

• Model size and architecture. A sparse mixture-of-experts model activates a fraction of its parameters per token; a dense model of similar total size does not. These have genuinely different energy profiles per query even at similar output quality.
• Batch size at serving time. Serving many requests together shares fixed overhead across them; a lightly-loaded server answering one request at a time is far less energy-efficient per query than the same hardware serving a full batch.
• Marginal vs. average grid carbon intensity. Whether you attribute a query’s energy to the average electricity mix of its region, or to the marginal generation source that actually ramped up to meet that demand, can change the resulting carbon estimate substantially, even for identical energy consumption.
• What’s included in the boundary. Some estimates count only the GPU’s draw; others include the rest of the server, networking, and cooling overhead (captured by PUE), which can roughly double or more the number depending on the facility.

None of this means the concern is overblown — it means precision beyond “small per query, large in aggregate, and growing” should be treated skeptically unless the source discloses its methodology.

What Practitioners Can Actually Do

For a small team running its own inference infrastructure rather than a hyperscaler, there are concrete, unglamorous levers that add up:

Cache aggressively. A meaningful share of real-world queries are near-duplicates or exact repeats. A semantic cache in front of the model turns a repeated question into a lookup instead of a fresh generation.

def answer(query, cache, model, similarity_threshold=0.95):
    cached = cache.lookup_similar(query, threshold=similarity_threshold)
    if cached is not None:
        return cached  # no inference call at all
    result = model.generate(query)
    cache.store(query, result)
    return result

Right-size the model to the task. Routing simple, high-volume queries to a small model and reserving a large model for genuinely hard requests cuts aggregate inference cost without touching quality on the queries that need the big model.

def route(query, small_model, large_model, complexity_classifier):
    if complexity_classifier.is_simple(query):
        return small_model.generate(query)
    return large_model.generate(query)

Batch requests where latency budget allows. Serving frameworks that queue and batch concurrent requests get substantially better throughput per watt than serving each request in isolation; this is a serving-infrastructure decision, not a model decision, and it’s often left on the table by default configs.

Pick your region and provider deliberately. Cloud regions differ enormously in grid carbon intensity depending on the local energy mix, and some providers publish per-region carbon or PUE figures. All else equal, that’s a free lever: same workload, meaningfully different footprint, just by choosing where it runs.

# Illustrative: preferring a lower-carbon-intensity region for batch/non-latency-critical inference
inference_jobs:
  batch_summarization:
    preferred_regions: ["region-with-high-renewable-share"]
    fallback_regions: ["region-b", "region-c"]
    max_latency_ms: null   # not latency-sensitive, free to schedule for lower-carbon windows

Cut wasted inference-time compute. Uncapped retry loops, agents that re-run failed steps without a budget, and speculative multi-sample generation that discards most samples are all inference cost with no corresponding value delivered. A step budget or a retry cap (see the agent-loop pattern used elsewhere in agentic tooling) is as much an energy-efficiency measure as a cost-control one.

Lever	Reduces	Effort to implement
Semantic caching	Inference (repeated queries)	Low
Model routing by task complexity	Inference (aggregate)	Medium
Serving-time batching	Inference (per-query efficiency)	Medium, mostly infra config
Region/provider selection	Grid carbon intensity of the same workload	Low, if regions are available
Retry/step budgets in agents	Wasted inference	Low

Key Takeaways

• Training is a large, one-time cost; inference is small per query but dominates lifetime footprint for any popular, long-lived model.
• Embodied cost from manufacturing the hardware itself is real and easy to overlook because it never appears on an energy bill.
• Water use in data centers is overwhelmingly for cooling, not consumption, and the technology (evaporative towers vs. closed-loop) drives most of the variance between facilities.
• Per-query energy and water figures vary by an order of magnitude or more across sources because of unstated methodology differences: model architecture, batch size, marginal vs. average grid mix, and system boundary.
• Treat precise per-query kWh or liter figures with skepticism unless the methodology is disclosed; the well-established fact is the direction and rough order of magnitude, not the exact number.
• Practical levers that actually reduce footprint: semantic caching, routing simple queries to smaller models, serving-time batching, deliberate region/provider selection, and capping retry/agent loops.
• None of these levers require waiting on a vendor or a regulator; they’re available to any team running its own inference stack today.