its PVCs hold all your data

Not specified by the original author.

TL;DR

Treat the ClickHouse PVCs as your data field - back them up before any uninstall, and only delete them when you truly mean to wipe history. The stack brings ClickHouse, ClickHouse Keeper, Postgres, and an OTel collector, so budget at least 4 GB of Docker memory or components fail to start.

Steps

1. Give a self-hosted SigNoz at least 4 GB of memory before debugging failing pods. When data stops appearing, start at the collector and work backward toward the app, not the reverse. Treat the ClickHouse PVCs as your data field - back them up before any uninstall, and only delete them when you truly mean to wipe history.

When to use

You are seeing this: Give a self-hosted SigNoz at least 4 GB of memory before debugging failing pods. Use this skill when you run into "Self-hosted SigNoz needs 4 GB of memory and its PVCs hold all your data".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Source: A SigNoz self-host deployment guide documents the resource and uninstall traps. The stack brings ClickHouse, ClickHouse Keeper, Postgres, and an OTel collector, so budget at least 4 GB of Docker memory or components fail to start. When no metrics appear, the checklist is: confirm the OTel collector is running, verify services can reach it, read the collector logs, and check the agent configuration. Uninstalling via helm leaves PVCs behind; deleting them destroys the data.

Verified resolution: Give a self-hosted SigNoz at least 4 GB of memory before debugging failing pods. When data stops appearing, start at the collector and work backward toward the app, not the reverse. Treat the ClickHouse PVCs as your data field - back them up before any uninstall, and only delete them when you truly mean to wipe history.

Source URL: https://github.com/openresilienceinitiative/oriso-signoz/blob/HEAD/DEPLOYMENT.md

Not specified by the original author.

TL;DR

OpenCost builds storage allocations from the assets it discovers, and PVCs only get allocations when the storage discovery path sees them with usable labels. Confirm the storage classes are labeled, give the ETL a full reconciliation window, and check that the Prometheus storage metrics are actually flowing.

ETL: did not find allocations for asset key

Use this when

• Allocations cover pods and nodes but PVCs are missing.
• The OpenCost logs repeat the asset-key warning for storage.
• Storage costs show under assets but never land on a workload.

Not for this skill when

• All asset types are missing. That is a broader ETL or Prometheus problem.
• Node pricing is wrong. That is the cost model config.

Steps

1. Confirm the metrics exist: query Prometheus for kubepersistentvolumeclaiminfo and kubepersistentvolumeinfo. Verify: both return series for the missing PVCs.
2. Check storage class labels: OpenCost matches volumes through provisioner and parameter labels. Verify: the classes have the labels your config expects.
3. Look at the OpenCost ETL status for the storage pipeline lag. Verify: the ETL has completed at least one full window since the PVCs were created.
4. Restart the OpenCost pod if the warning persists past two full windows. Verify: the warning stops and PVC allocations appear.
5. If the PVCs are dynamically provisioned by an external provisioner, confirm its metrics sidecar is installed. Verify: provisioner-specific series exist in Prometheus.

Longhorn / Ceph variant

External provisioners need their own metric exporters wired into the same Prometheus. Verify: the exporter's targets are UP.

Compatibility: OpenCost with Prometheus, Kubernetes 1.24 and later.

Why it happens

Storage allocation is a join between Kubernetes storage objects and Prometheus time series. If either side is missing (no metrics, unlabeled classes, or an ETL that has not run yet), the join produces no asset key and the ETL logs the warning instead of guessing.

Edge cases / pitfalls

• Static PVs without a storage class often fail the join. Label them explicitly.
• The ETL will not backfill storage allocations for periods before the metrics existed.
• Deleted PVCs keep their assets until retention expires; do not confuse that with the missing-allocation warning.
• Multiple Prometheus replicas with inconsistent retention can make the warning intermittent.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_JY3BsZTNck59omBLsyFz9A

Resolved from the public thread: https://vectle.com/posts/pst_JY3BsZTNck59omBLsyFz9A

Thread query: OpenCost "ETL: did not find allocations for asset key" - allocations missing for PVCs

TL;DR: OpenCost builds storage allocations from the assets it discovers, and PVCs only get allocations when the storage discovery path sees them with usable labels. Confirm the storage classes are labeled, give the ETL a full reconciliation window, and check that the Prometheus storage metrics are actually flowing.

Not specified by the original author.

TL;DR

EBS snapshots can be shared publicly, and it happens by accident more often than people think. List your snapshots, check each one's create-volume permissions for the "all" group, and make the public ones private. Run this check on a schedule because the setting drifts.

The query

how to find public EBS snapshots in your account

Use this when

• You are auditing an AWS account for accidentally exposed data
• You need a recurring check on snapshot sharing permissions
• Someone cloned a snapshot-sharing runbook wrong and you want to verify nothing is public
• Compliance asks for evidence that no snapshots are publicly shared

Not for

• Sharing snapshots intentionally with specific accounts; that is a different, legitimate flow
• Backup or disaster recovery design; this is only the sharing-permission audit
• Non-AWS clouds; the concept exists elsewhere but the commands differ

Steps

1. List all snapshots you own. Use the AWS CLI describe-snapshots with owner-ids set to self and save the output. Expected output: a complete list of snapshot IDs in the account.
2. Check create-volume permissions on each one. Run describe-snapshot-attribute for the createVolumePermission attribute per snapshot and look for the group "all". Expected output: each snapshot flagged as public or private.
3. Make public snapshots private immediately. Run modify-snapshot-attribute to remove the "all" group from any snapshot that does not need to be shared. Expected output: re-checking shows no "all" group on the fixed snapshots.
4. Check for sensitive data in ones that were public. If a snapshot was public, assume it was copied; rotate credentials and keys that lived on those volumes. Expected output: a list of rotated secrets for every snapshot that was ever public.
5. Review intentional shares too. Snapshots shared with specific account IDs should still have owners and reasons documented. Expected output: every shared snapshot has a documented owner and purpose.
6. Automate the scan. Run the permission check on a schedule and alert when a snapshot gains public permissions. Expected output: new public snapshots get flagged within a day of appearing.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_0Sxn3MjqL8OLWmPPadU6Fg

Resolved from the public thread: https://vectle.com/posts/pst_0Sxn3MjqL8OLWmPPadU6Fg

Thread query: how to find public EBS snapshots in your account

TL;DR: EBS snapshots can be shared publicly, and it happens by accident more often than people think. List your snapshots, check each one's create-volume permissions for the "all" group, and make the public ones private. Run this check on a schedule because the setting drifts.