Huge data loss when deploying refinery configuration without restarting

TL;DR

Deploying new refinery.toml/rules.toml without restarting the refinery systemd service causes almost total trace loss: traces making it to Honeycomb drop to near zero after config deploy. Switching the deployment script to cp instead of File.write fixed it, and trace volume returned to normal in the graphs.

When to use

You are seeing this: The service keeps running and live-reload picks up the new rules. Use this skill when you run into "Huge data loss when deploying refinery configuration without restarting".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Steps

1. Deploying new refinery.toml/rules.toml without restarting the refinery systemd service causes almost total trace loss: traces making it to Honeycomb drop to near zero after config deploy. The service keeps running and live-reload picks up the new rules. Deployment script wrote configs with Ruby's File.write then called systemctl start (a noop when already running).
2. The reporter tracked the data loss down to how the config files were written: Ruby's File.write modifies the file in place while refinery's live-reload reads it. Switching the deployment script to cp instead of File.write fixed it, and trace volume returned to normal in the graphs. If your live-reload config deploys are mysteriously breaking sampling, try switching the write to an atomic replacement and see whether the loss disappears.
3. Take the question above and check if it matches what you're seeing. If it does, work through the verified fix step by step. Start with the cause described first, because that's what tells you the fix applies to your setup, then apply the changes in the order given. Verify by re-running whatever failed before, and expect the same behavior the thread author reported.

Someone asked: Deploying new refinery.toml/rules.toml without restarting the refinery systemd service causes almost total trace loss: traces making it to Honeycomb drop to near zero after config deploy. The service keeps running and live-reload picks up the new rules. Deployment script wrote configs with Ruby's File.write then called systemctl start (a noop when already running). The verified answer: The reporter tracked the data loss down to how the config files were written: Ruby's File.write modifies the file in place while refinery's live-reload reads it. Switching the deployment script to cp instead of File.write fixed it, and trace volume returned to normal in the graphs. If your live-reload config deploys are mysteriously breaking sampling, try switching the write to an atomic replacement and see whether the loss disappears.

Source: https://github.com/honeycombio/refinery/issues/384 Discussion and confirmation live at https://vectle.com/threads/thr_WC6CkdnlyCEMYKPgYbhBFQ