Fix "Memory limit (total) exceeded"
Diagnose ClickHouse's server-wide "Memory limit (total) exceeded" error — check max_server_memory_usage, concurrent query load, and cache sizes.
Code: 241. DB::Exception: Memory limit (total) exceeded: would use X GiB (attempt to allocate chunk of Y bytes), maximum: Z GiB looks like Memory limit (for query) exceeded, but the (total) wording matters: this is the server-wide cap (max_server_memory_usage, defaulting to roughly 90% of physical RAM), tripped by every concurrent query, merge, and cache combined — not one query's own budget. It can fire even when no single query looks expensive.
Why it happens
- Too many concurrent queries. Each one uses a moderate amount of memory, but the sum crosses the server cap.
- Oversized caches.
mark_cache_sizeanduncompressed_cache_sizeare configured too large relative to available RAM. - Merges and mutations competing for memory alongside queries — a merge storm (see Merges slower than inserts) or a large
ALTER … UPDATE/DELETEadds to the same budget. - Replication buffers on a busy replica queueing a lot of in-flight data.
Diagnose
Check current and historical server-wide memory usage, then find what's consuming it:
-- Current total memory tracked by the server
SELECT value AS current_memory_bytes, formatReadableSize(value) AS current_memory
FROM system.metrics
WHERE metric = 'MemoryTracking';-- Memory usage trend over the last few hours (asynchronous_metrics is sampled periodically)
SELECT event_time, value AS memory_bytes
FROM system.asynchronous_metric_log
WHERE metric = 'MemoryTracking' AND event_time > now() - INTERVAL 6 HOUR
ORDER BY event_time;-- Everything currently holding memory, heaviest first
SELECT
query_id,
user,
elapsed,
formatReadableSize(memory_usage) AS current_memory,
query
FROM system.processes
ORDER BY memory_usage DESC
LIMIT 20;Confirm the configured ceiling with SELECT * FROM system.server_settings WHERE name = 'max_server_memory_usage' (ClickHouse 24.x+), or check max_server_memory_usage / max_server_memory_usage_to_ram_ratio in config.xml / config.d/*.xml on older versions.
Fix
- Limit concurrency — cap
max_concurrent_queriesand per-usermax_memory_usage_for_userso no single user or workload can consume the whole budget. - Shrink caches if they're oversized for the box —
mark_cache_size,uncompressed_cache_size, and the primary-index cache all count against the same total. - Raise
max_server_memory_usageif the host genuinely has spare RAM the default 90% ratio isn't using. - Spread load across more replicas or shards instead of concentrating queries on one node.
- Address merge/mutation pressure if that's the contributor — see Too many parts and Merges slower than inserts.
Don't just raise the ceiling
Raising max_server_memory_usage without addressing runaway concurrency risks the Linux OOM killer taking down the whole clickhouse-server process instead of one query failing cleanly — a much worse outage. See Altinity's "Rescuing ClickHouse from the Linux OOM Killer" for the failure mode this is protecting against.
See it in chmonitor
The Metrics page tracks MemoryTracking and other server counters live, and Health rolls memory pressure into the at-a-glance status grid so you see it trending up before the server rejects a query. Try it on the live demo.
Related
Fix "Memory limit (for query) exceeded"
Diagnose ClickHouse MEMORY_LIMIT_EXCEEDED errors — find the offending query in system.query_log, tune max_memory_usage, and enable external aggregation.
Upgrading ClickHouse
Safely upgrade ClickHouse when chmonitor is connected — pre-upgrade checks, version-by-version dashboard changes, and post-upgrade validation.