agent reported '1200 queries per request' but the trace included its own instrumentation queries

TL;DR

The agent counted its own homework. Profiler instrumentation (metric scrapes, health checks, the profiler's own queries against its schema) runs inside the traced window and gets tallied as application queries. Fingerprint the instrumentation queries, exclude them with a connection-level tag, and recount. The real number is usually a fraction of the reported one.

The query

agent reported '1200 queries per request' but the trace included its own instrumentation queries

Steps

1. Dump the full query list behind the count

Get every query the agent counted for the request: SQL text, timestamp, and connection metadata (database user, application name, client address).

Expected: the full list of queries the 1200 number was built from.

2. Fingerprint the agent's own queries

Look for the profiler's signature: queries against its own schema or metrics tables, periodic health checks, and queries issued by the profiler's database user or connection. Polling patterns (same query every N seconds) are a giveaway.

Expected: a clearly identifiable subset of queries belonging to the instrumentation, not the application.

3. Re-run with instrumentation tagged and excluded

Tag the profiler's connections (set the application name to something identifiable, or use a dedicated database user for the agent) and filter tagged queries out before counting. Re-run the same request.

Expected: the count drops to the real application number, and the excluded set matches the fingerprinted subset from step 2.

4. Verify the remainder is plausible

Sanity-check the remaining count against the request's actual work: number of endpoints hit, associations loaded, cache lookups. If it is still surprisingly high, there may be a genuine issue underneath the instrumentation noise.

Expected: a believable per-request count consistent with the code path.

5. Make the exclusion the default

Change the detector to exclude its own queries by default (filter on the instrumentation's application name or user) and to document the exclusion in its output. No trace analysis should ever count the observer.

Expected: future runs report the application count with a note like "N instrumentation queries excluded."

Use this when

• A per-request query count looks impossibly high
• The trace contains the profiler's own schema or metric queries
• Counts vary with the profiler's polling interval
• The agent and the app share database connections

Not for this skill when

• The queries are all genuine application queries (real N+1)
• The duplicates are retries after deadlocks
• The count comes from log-line miscounting rather than trace pollution
• The "duplicates" differ in WHERE clauses

Variant phrasings

query count includes monitoring queries

Same fix: identify by connection metadata and exclude before counting.

profiler counted its own queries

The observer effect in its purest form. Tag the observer's traffic.

per-request count drops when profiler is off

Confirms the instrumentation was in the count. Keep the profiler on, exclude its queries instead.

Why it happens

Tracers observe at the connection or database level, where every query looks the same regardless of who issued it. The profiler's own queries flow through the same pipes as the application's, and nothing in the SQL text says "this one is mine." The agent built its count from the raw stream and never asked which queries belonged to the thing doing the measuring.

Edge cases

• Shared database user: if the app and the profiler connect as the same user, the user column cannot separate them. Use the application name or a comment marker on the profiler's queries instead.
• Async metric flushes: instrumentation queries can land mid-request even when the profiler "isn't running" a trace. Tagging must be on the connection, not the trace window.
• Sampling profilers: some profilers periodically query pgstatstatements itself. Those reads are instrumentation too.
• Legitimate high counts: after exclusion the number may still be high. Do not assume every surprising count is pollution; verify the remainder (step 4).

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_fHOExct6RLzNPgnrGxp02A