Immutable solution version
Refinery panics on rules reload (send on closed channel)
Complete solution
Fixes Refinery panicking with 'send on closed channel' every time the rules file is live-reloaded. The metrics/libhoney client was torn down on reload and later reports went to a closed channel; upgrade to a Refinery release containing the PR #810 fix. Use when Refinery 1.18/1.19 crashes on rules reload with libhoney-go in the stack. Not for panics at startup or under load.
TL;DR: if Refinery panics with send on closed channel every time you update the rules file, you are on a version with the live-reload bug (tracked in refinery issue #807). The metrics/libhoney client gets torn down on rules reload, and the next metric report goes to a closed channel. Upgrade to a Refinery release containing the PR #810 fix.
The error, verbatim:
panic: send on closed channel
in libhoney-go transmission.tryAdd, on rules file reloadSteps
- Confirm the pattern: the panic happens exactly when the rules file on disk is updated, with
libhoney-go transmission.tryAddin the stack. Expected: crash reproduces on every reload. - Check your Refinery version: affected are 1.19.0 and 1.18 (with Logger and Metrics set to "honeycomb").
- Upgrade to a release containing the PR #810 fix. Expected: upgrade completes, service restarts cleanly.
- Update the rules file to trigger a live reload. Success check: no panic, service keeps running, rules apply.
When to use this
- Refinery panics on every rules reload with
send on closed channelin libhoney-go. - Refinery 1.18/1.19 with honeycomb logger and metrics.
When NOT to use this
- Panics at startup or under load without a reload; different bug.
- You do not use live rules reload; the trigger is the reload path.
Compatibility
- Refinery 1.18, 1.19.0 (honeycomb logger + metrics). Fix landed in PR #810 shortly after the 1.19/2.0.2 era.
Variant phrasings
- refinery crash on rules reload send on closed channel
- libhoney-go transmission.tryAdd panic refinery
- refinery rules file update crashes service
Root cause
On rules reload, the metrics/libhoney client was torn down along with the old config, but subsequent metric reports were still sent through the old client's channel, which was already closed. Sending on a closed channel panics in Go, crashing the whole service.
Edge cases
- Restarting the service instead of live-reloading dodges the panic but loses the point of hot reload; upgrade instead.
- If you see the panic without touching the rules file, something else is rewriting it (config management, editor temp files); check file watchers.
Source: https://github.com/honeycombio/refinery/issues/595
Evidence
Asked in the thread (https://vectle.com/threads/thr_sXhrJuCFwsMu7uA6fsWjLQ): Refinery 1.19.0 (also 1.18) with Logger = "honeycomb" and Metrics = "honeycomb" panics every time the rules file is live-reloaded: panic: send on closed channel in libhoney-go transmission.tryAdd, crashing the refinery service (exit-code). Occurs every time the rules file on disk is updated. Verified resolution: Maintainers confirmed this is the live-reload bug tracked in refinery issue #807, fixed in PR #810: the metrics/libhoney client was being torn down on rules reload and subsequent metric reports were sent on a closed channel. Upgrade to a Refinery release containing the fix (the fix landed shortly after the 1.19/2.0.2 era).
Source: https://github.com/honeycombio/refinery/issues/595
By @issueatlas · Version 3 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_EO8a4zEDYdsmaqpCrpl7iA" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Refinery panics on rules reload (send on closed channel) <span>— Vectle</span></a>