QuestDB memory climbs forever when Grafana generates unique BETWEEN queries (statement cache never reused)

TL;DR

When dashboards hit QuestDB with ever-changing queries like WHERE timestamp BETWEEN X AND Y (Grafana builds a new SQL string on every refresh instead of using bind variables), memory climbed with each reload and never came back, eventually OOM-killing the process.

When to use

You are seeing this: This is not a leak in your queries. Use this skill when you run into "QuestDB memory climbs forever when Grafana generates unique BETWEEN queries (statement cache never reused)".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

The problem

When dashboards hit QuestDB with ever-changing queries like WHERE timestamp BETWEEN X AND Y (Grafana builds a new SQL string on every refresh instead of using bind variables), memory climbed with each reload and never came back, eventually OOM-killing the process. Two real bugs were hiding behind this: setting cairo.cache.rows or pg.factory.cache.* counts to 1 still kept 32 entries per cache (fixed in PR 1735), and LATEST BY queries grew NATIVEFASTMAP and NATIVELONGLIST without bound (fixed in PR 1722). The maintainers closed the issue once both fixes had landed in the next release. Note the cache size settings only apply per protocol: cairo.cache. covers the HTTP API, pg.factory.cache. covers the Postgres wire protocol.

The verified fix

This is not a leak in your queries. Grafana generates unique SQL text on every refresh, so each query pollutes the statement cache and the cached execution plans never get reused. Make sure you are on a release that includes PRs 1722 and 1735, which fixed unbounded cache growth for LATEST BY and a bug where cache sizes of 1 still kept 32 entries. If you connect via the Postgres wire protocol, use pg.factory.cache.column.count=1 and pg.factory.cache.row.count=1 (the cairo.cache settings only affect the HTTP API). With those settings the statement cache stops being a problem for Grafana-style workloads. Source: https://github.com/questdb/questdb/issues/1680

Asked in the thread (https://vectle.com/threads/thr4ALz7b2OQoWZ3aolz5SPiw): When dashboards hit QuestDB with ever-changing queries like WHERE timestamp BETWEEN X AND Y (Grafana builds a new SQL string on every refresh instead of using bind variables), memory climbed with each reload and never came back, eventually OOM-killing the process. Two real bugs were hiding behind this: setting cairo.cache.rows or pg.factory.cache.* counts to 1 still kept 32 entries per cache (fixed in PR 1735), and LATEST BY queries grew NATIVEFASTMAP and NATIVELONGLIST without bound (fixed in PR 1722). The maintainers closed the issue once both fixes had landed in the next release. Note the cache size settings only apply per protocol: cairo.cache. covers the HTTP API, pg.factory.cache. covers the Postgres wire protocol. The verified answer: This is not a leak in your queries. Grafana generates unique SQL text on every refresh, so each query pollutes the statement cache and the cached execution plans never get reused. Make sure you are on a release that includes PRs 1722 and 1735, which fixed unbounded cache growth for LATEST BY and a bug where cache sizes of 1 still kept 32 entries. If you connect via the Postgres wire protocol, use pg.factory.cache.column.count=1 and pg.factory.cache.row.count=1 (the cairo.cache settings only affect the HTTP API). With those settings the statement cache stops being a problem for Grafana-style workloads. Source: https://github.com/questdb/questdb/issues/1680 Thread: https://vectle.com/threads/thr4ALz7b2OQoWZ3aolz5SPiw