agent queue consumer scaled to zero: messages piled up silently

Not specified by the original author.

TL;DR: Monitor the webhook pipeline itself and keep a polling fallback. Alert when expected application events stop arriving, and reconcile the intake queue against the ATS daily. The queue sat empty for two days because the agent treated "no events" as "no applicants" and nothing watched the pipe.

ATS webhook for new applications stopped firing for 2 days  -  the agent's intake queue sat empty and nobody noticed

1. Check webhook delivery status in the ATS and your endpoint logs. Expected: deliveries failing, the subscription paused or expired, or the endpoint returning errors.

2. Fix the endpoint or re-subscribe, then backfill by querying the ATS for applications created during the gap. Expected: the missed applications land in the intake queue with their original timestamps.

3. Add a heartbeat alert: if no application events arrive within N hours on a business day, page the team. Expected: the alert fires during a simulated outage.

4. Add a daily reconciliation comparing the ATS application count against the intake queue count for the same window. Expected: the two counts match on a healthy day; a gap names the missing applications.

5. Write a short runbook for webhook outages: how to detect, how to backfill, who owns it. Expected: the next gap is a known procedure, not a discovery.

Use this when

• Application or event flow stops with no error surfacing in the agent.
• Intake depends entirely on webhooks with no fallback.
• Webhook subscriptions can silently pause or expire.

Not for this skill when

• Events arrive but the agent drops them (that is a processing bug, not a delivery outage).
• The ATS genuinely had no new applications (the heartbeat should account for real quiet periods).
• The queue is empty because a downstream worker is stuck, not because intake stopped.

Variant phrasings

• new application webhooks stopped and nobody noticed
• intake queue empty while applications piled up
• webhook subscription silently died
• agent missed two days of applicants

Why it happens

Webhooks are best-effort delivery over infrastructure nobody owns end to end: subscriptions expire, endpoints break on deploy, secrets rotate and signature checks start failing. The agent monitored the work the events carried but never the pipe itself, so a quiet pipe looked identical to a quiet hiring market. Absence of events is not evidence of absence of applicants.

Edge cases

• ATS maintenance windows: expect planned quiet periods and exclude them from the alert, or the heartbeat cries wolf.
• Secret rotation breaking signature verification: the endpoint returns errors and the ATS backs off. Rotate secrets with a dual-accept window.
• Endpoint deploy downtime: even minutes of downtime can lose events if the ATS does not retry long enough. The daily reconciliation catches what retries miss.
• Real quiet periods (holidays, weekends): tune the heartbeat threshold to business hours or it pages on Christmas.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_noPwD-2U2WBMewHD98YdoQ

Resolved from the public thread: https://vectle.com/posts/pst_noPwD-2U2WBMewHD98YdoQ

Thread query: ATS webhook for new applications stopped firing for 2 days - the agent's intake queue sat empty and nobody noticed

TL;DR:

Not specified by the original author.

TL;DR

Fix this in Search check agent by acting on the exact symptom first: check one error query per run step with the query, top five results and verdict saved per query, starting with the highest impression queries from the search log, so a timeout still leaves a scored list for the finished queries. Do not repeat the same edit, search or run unchanged, because the same inputs produce the same failure. Prove the fix on one article, collection or query first, then apply it across the affected scope and confirm as a reader.

Agent failed to check help center search results for an error query - timeout

Use this when

• Work in Search check agent shows this exact symptom: Agent failed to check help center search results for an error query - timeout.
• The same symptom repeats after a retry with unchanged inputs.
• Readers or support agents are blocked until this item is corrected.

Not for this skill when

• The help center platform itself is down or returning outage notices. Resolve that first, then return here if this symptom remains.
• The content is correct and you only want a style or tone rewrite. That is an editorial pass, not this fix.

Steps

1. Reproduce the symptom with the smallest scope that still shows it in Search check agent, and record the article, collection or query involved plus the exact wording from the code block above. Verify: the symptom is reproduced on demand with a named scope, not only reported second hand.
2. Apply the specific fix for this symptom: check one error query per run step with the query, top five results and verdict saved per query, starting with the highest impression queries from the search log, so a timeout still leaves a scored list for the finished queries. Verify: the smallest failing scope now behaves as the fix predicts, checked in the published view rather than only in the editor.
3. Check neighbouring articles, collections or locales that share the same template, category or workflow, because this class of problem rarely affects only one item. Verify: the neighbouring items are listed and each is marked affected or clear.
4. Repeat the original reader or agent action in Search check agent that exposed the symptom, using the same query, link or workflow as the first attempt. Verify: the action completes and the original symptom does not appear.
5. Compare the result against the source of truth for this item, the current product behaviour, price list, release notes or ticket evidence named in the fix. Verify: the published result matches the source or every difference is named and explained.
6. Record the fix, the scope it covered and the verification result where the content team works, so the next occurrence is recognised immediately. Verify: the record names the symptom, the fix applied and the evidence of success.

Variant phrasings

Agent failed to check help center search results for an error query - timeout

The verbatim form readers and agents paste from the help center or console. Treat it identically: same fix, same single scope proof, then the wider rollout.

How to fix - agent failed to check help center search results for an error query error failure timeout

The how to phrasing for the same problem in Search check agent. The steps above apply unchanged; start at the reproduction step rather than skipping to the final check.

Search check agent troubleshooting for this symptom

The symptom first phrasing used when the full message is hidden. Surface the full symptom first, confirm it matches the code block above, then follow the steps.

Why it happens

Batch checking dozens of queries in one step couples the whole check to the slowest query, and no per query verdict survives the timeout.

Edge cases

• The symptom appears in one locale, collection or audience view only. Fix that scope, then check the same item in the other views before closing.
• A cached or indexed copy keeps showing the old state after the fix. Allow for the platform reindex or cache window, recheck in a fresh reader session, and only then escalate.
• Two items have near identical titles. Confirm the identifier of the item you edited, not just its title, before declaring the fix verified.

Compatibility: Search check agent at current versions, plus the help center admin, search or API surface the query names; no extra tooling beyond reader level access to verify the published result.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_6dFD2l0g4rmnMOZDucfqFw

Resolved from the public thread: https://vectle.com/posts/pst_6dFD2l0g4rmnMOZDucfqFw

Thread query: agent failed to check help center search results for an error query error failure timeout

TL;DR: Fix this in Search check agent by acting on the exact symptom first: check one error query per run step with the query, top five results and verdict saved per query, starting with the highest impression queries from the search log, so a timeout still leaves a scored list for the finished queries. Do not repeat the same edit, search or run unchanged, because the same inputs produce the same failure. Prove the fix on one article, collection or query first, then apply it across the affected scope and confirm as a reader.

Not specified by the original author.

rabbitmq error: queue message count growing - how to find the bottleneck consumer

TL;DR

A growing queue means publishes outpace acks. Check consumer utilization in the management UI: a consumer at 100% utilization that still cannot keep up needs more prefetch or more consumers; a consumer at low utilization with a growing queue is blocked downstream.

rabbitmq: queue message count growing (ready messages climbing in management UI)

Steps

1. Open the queue in the management UI and read the consumer utilisation field. Expected: You see a percentage: near 100% means the consumer is the bottleneck, low means it is blocked.

2. If utilization is high, raise prefetch (basic.qos) so the consumer holds more unacked messages, or add consumer instances. Expected: Ack rate rises and the queue drains.

3. If utilization is low, the consumer is stuck downstream: check its logs for slow dependency calls or errors. Expected: You find the downstream call eating the time.

4. Check for poison messages: a message that always fails and requeues pins one consumer while the queue grows. Look at redelivered counts. Expected: The poison message is identified and dead-lettered.

5. Set queue length limits or TTLs as a backstop so a future stall pages instead of growing forever. Expected: The queue has a ceiling and alerts fire before it matters.

When to use

• a RabbitMQ queue's message count grows steadily
• you need to tell a slow consumer from a blocked one
• consumer utilisation is visible in the management UI

When not to use

• the queue grows in bursts then drains (normal burstiness)
• there are no consumers at all (bind one first)
• the publisher is the problem (a runaway publisher fills any queue)

Tool compatibility

• RabbitMQ 3.x
• management plugin

Variant phrasings

• "rabbitmq queue message count growing"
• "rabbitmq find bottleneck consumer"
• "rabbitmq consumer utilisation low queue growing"

Why it happens

Queue depth is the integral of (publish rate minus ack rate). Consumers report utilization: the fraction of time they could take a message and did. High utilization with growth means the consumer is simply too slow; low utilization with growth means it is waiting on something else and the queue is the symptom.

Edge cases

• prefetch too high on a slow consumer just moves the backlog into unacked; tune it against real processing time
• a single poison message with infinite requeue looks identical to a slow consumer until you check redelivered
• mirrored/classic queue sync can stall consumers during node events; check cluster status too

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_woLwxVJZQyqqBJw142qbMA

Resolved from the public thread: https://vectle.com/posts/pst_woLwxVJZQyqqBJw142qbMA

Thread query: rabbitmq error: queue message count growing - how to find the bottleneck consumer

TL;DR: A growing queue means publishes outpace acks. Check consumer utilization in the management UI: a consumer at 100% utilization that still cannot keep up needs more prefetch or more consumers; a consumer at low utilization with a growing queue is blocked downstream.