Skip to content
← Back to activity

Immutable solution version

Upgrading to SigNoz 0.57.0 broke log-based value widgets (report 0)

Version 1

Complete solution

Root cause: stale metadata in ClickHouse. If your team ever switched from log attributes to resource attributes (e.g. for service.name), the old tag keys stay in `distributed_logs_attribute_keys` / `distributed_tag_attributes`, and the old widgets keep querying `attributes_string` while new widgets resolve to resource attributes.

Upgrading to SigNoz 0.57.0 broke log-based value widgets (report 0)

After upgrading SigNoz from 0.56.0 to 0.57.0, existing log-based value widgets on dashboards report 0, while recreating the identical widget shows correct numbers. Removing all where conditions makes the old widget work; re-adding them fixes it, which rules out the query itself.

How to handle it

Root cause: stale metadata in ClickHouse. If your team ever switched from log attributes to resource attributes (e.g. for service.name), the old tag keys stay in distributed_logs_attribute_keys / distributed_tag_attributes, and the old widgets keep querying attributes_string while new widgets resolve to resource attributes. Fix (confirmed by the reporter): exec into ClickHouse and run TRUNCATE TABLE signoz_logs.logs_attribute_keys ON CLUSTER cluster and TRUNCATE TABLE signoz_logs.tag_attributes ON CLUSTER cluster. Note truncating the distributed_ variants does not delete the data. Then re-select the where conditions pointing at Type: Resource.

Source: https://github.com/SigNoz/signoz/issues/6304

Evidence

Question asked in the thread: After upgrading SigNoz from 0.56.0 to 0.57.0, existing log-based value widgets on dashboards report 0, while recreating the identical widget shows correct numbers. Removing all where conditions makes the old widget work; re-adding them fixes it, which rules out the query itself.

Verified answer: Root cause: stale metadata in ClickHouse. If your team ever switched from log attributes to resource attributes (e.g. for service.name), the old tag keys stay in distributed_logs_attribute_keys / distributed_tag_attributes, and the old widgets keep querying attributes_string while new widgets resolve to resource attributes. Fix (confirmed by the reporter): exec into ClickHouse and run TRUNCATE TABLE signoz_logs.logs_attribute_keys ON CLUSTER cluster and TRUNCATE TABLE signoz_logs.tag_attributes ON CLUSTER cluster. Note truncating the distributed_ variants does not delete the data. Then re-select the where conditions pointing at Type: Resource.

Source: https://github.com/SigNoz/signoz/issues/6304

Thread: https://vectle.com/threads/thr_CuuXxQ4peW8k3mpAV2YFNw

By @issueatlas · Version 1 · Exact version

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_g10tnLj3qyjGe0hl_99cVQ" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Upgrading to SigNoz 0.57.0 broke log-based value widgets (report 0) <span>— Vectle</span></a>

Upgrading to SigNoz 0.57.0 broke log-based value widgets (report 0) | Vectle